aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-27 20:25:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

1022 commits

This branch
This branch
All branches
Author SHA1 Message Date
Andrej Karpathy
60a1037db9 Merge branch 'ChrisDryden-layernorm_backwards_mem_saving' 2024-05-31 22:04:44 +00:00
Andrej Karpathy
f4874a9f63 make things a bit more consistent, keep recompute 1 as default, and add a comment 2024-05-31 22:04:34 +00:00
Andrej Karpathy
629a24020d Merge branch 'layernorm_backwards_mem_saving' of https://github.com/ChrisDryden/llm.c into ChrisDryden-layernorm_backwards_mem_saving 2024-05-31 21:47:28 +00:00
Andrej Karpathy
ed37d9261b fix bug in readme 2024-05-30 21:39:47 +00:00
Andrej Karpathy
b48224cb0c refactor the README file because it was getting crazy 2024-05-30 21:39:06 +00:00
Andrej Karpathy
0fddf06043 more docs on scripts 2024-05-30 21:06:36 +00:00
Andrej Karpathy
fe36e40f2c also move pyrun script to scripts/ 2024-05-30 20:17:16 +00:00
Andrej
d98d3ed82b
code refactor, introduce llmc dir and scripts dir 
The main file train_gpt2.cu is getting out of control. So here we are.
 2024-05-30 13:13:04 -07:00
Andrej Karpathy
22d1607f6d delete these files, they are now in scripts/ folder and better 2024-05-30 20:11:59 +00:00
Andrej Karpathy
fc40ffc4aa adjust readme to new scripts/ dir 2024-05-30 20:10:04 +00:00
Andrej Karpathy
a0d5dbbda6 scripts folder 2024-05-30 20:00:07 +00:00
Andrej Karpathy
35aa110244 more refactor 2024-05-30 19:39:29 +00:00
Andrej Karpathy
f8f9e49ba3 Merge branch 'master' into feature/libdir 2024-05-30 19:07:51 +00:00
Andrej Karpathy
5f547601a0 further hide the c pytorch bridge logic 2024-05-30 18:20:29 +00:00
Andrej Karpathy
ca27eb8fa5 bugfix don't train on val batch 2024-05-30 18:18:05 +00:00
Christopher
59432fea19 modified the recalculation parameter to be based off of 2 instead of 1 2024-05-30 17:23:47 +00:00
Christopher
e5719e18b3 Rearranged layernorm forward for 1 and 2 and used layer variables instead. Validated with test file 2024-05-30 16:28:03 +00:00
Christopher
95b1fad088 First step in creating option to recalculate layernorm activations in backwards pass 2024-05-30 16:28:03 +00:00
Andrej
0e2298247a
Merge pull request #496 from ChrisDryden/patch-7 
Nit: duplicated line in header of project
 2024-05-30 09:13:30 -07:00
Andrej Karpathy
862c957d3a fix default number of gpus from 4 to 8 for consistency with run124M.sh 2024-05-30 14:50:09 +00:00
Andrej
ed805f60be
Merge pull request #494 from karpathy/feature/pytorch_parity 
pytorch parity v1
 2024-05-30 07:47:58 -07:00
Andrej Karpathy
5959851d2d small comments for pyrun124M.sh 2024-05-30 14:47:43 +00:00
Andrej Karpathy
9b935a8f04 fix bug with advancing position in dataloader 2024-05-30 13:16:07 +00:00
Chris Dryden
3bca31fde1
Nit: duplicated line in header of project 2024-05-29 20:03:13 -07:00
Christopher
5450632237 Removed unnecesary shared memory due to blockreduce using static defined shared memory 2024-05-30 03:01:52 +00:00
Andrej Karpathy
555d07502a more feature parity work, i think it's quite close now 2024-05-30 02:29:40 +00:00
Andrej Karpathy
ea88a343b5 also updated pyrun124M.sh 2024-05-30 02:05:41 +00:00
Andrej Karpathy
03c2e85601 sharded dataloader for PyTorch 2024-05-30 02:05:06 +00:00
Andrej Karpathy
3fdf88e4d4 add logfile similar to C 2024-05-29 23:59:35 +00:00
Andrej Karpathy
083c50a52e small stylistic touchups that match the c code 2024-05-29 23:50:13 +00:00
Andrej Karpathy
1f6fdfbd9a add pyrun124M.sh 2024-05-29 23:30:27 +00:00
Andrej Karpathy
c413bfe0ca pytorch parity v1 2024-05-29 23:25:55 +00:00
Andrej Karpathy
4cc8b1570b hardcode grad_clip to 1.0, delete confusion around legacy code that used -c flag for it, but became inactive in the if else block 2024-05-29 19:01:12 +00:00
Andrej
55f3665b64
Merge pull request #493 from vyom1611/master-2 
Modal benchmarking script updated to replace deprecated calls
 2024-05-29 07:50:10 -07:00
vyom1611
062f096da1 Modal benchmarking script updated to replace deprecated calls 2024-05-29 20:01:41 +05:30
Andrej
2db7716d68
Merge pull request #476 from lancerts/layernorm-bk-fix 
Fix "Mismatch of dweight at layernorm_backward.cu #428"
 2024-05-29 07:30:52 -07:00
Andrej
c3a65a556f
Merge pull request #487 from huoyushequ/patch-1 
`softmax_autoregressive_backward_kernel` does not use share memory in the kernel
 2024-05-28 18:52:38 -07:00
QIAOQIAN
27cf5223e0
softmax_autoregressive_backward_kernel does not use share memory in the kernel 
we do not need to launch the kernel with 256 bytes share memory, so remove it
 2024-05-29 09:23:46 +08:00
Erik Schultheis
daa95e04ae ampere-consumer 2024-05-29 02:27:11 +03:00
Erik Schultheis
de7a9b83cd more GPUs 2024-05-29 02:20:31 +03:00
Erik Schultheis
184cc9d10a added H100 SMX version 2024-05-29 01:31:16 +03:00
Erik Schultheis
e2b411362e include string view 2024-05-29 00:54:23 +03:00
Erik Schultheis
9374d751de MFU for other GPUs 2024-05-29 00:48:37 +03:00
Andrej Karpathy
a664aa3094 include datasets as req 2024-05-28 21:43:11 +00:00
Andrej
afe40be994
Merge pull request #479 from rosslwheeler/feature/ci-manual-trigger 
Adding manual trigger for CI
 2024-05-28 12:46:27 -07:00
Andrej Karpathy
88dd670c42 link to recent post from readme 2024-05-28 17:31:31 +00:00
Andrej Karpathy
a6a2f042f8 keep -y 1 flag 2024-05-28 15:24:30 +00:00
Andrej Karpathy
a7f01088dc repro scripts for 124M and 350M models 2024-05-28 15:08:04 +00:00
Andrej Karpathy
037ec67a5d small tweak to launch 2024-05-28 15:07:07 +00:00
Andrej Karpathy
0c83a23188 tweak the logfile parser and visualizer 2024-05-28 01:38:15 +00:00