aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-28 20:35:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

1022 commits

This branch
This branch
All branches
Author SHA1 Message Date
Andrej Karpathy
3de05baaf4 Merge branch 'ngc92-errchk' 2024-05-18 18:35:28 +00:00
Andrej Karpathy
4374360015 adjust wte upper bound a bit, and print always because this part is really tricky and i don't trust anything other than manual inspection, even if we pass, allegedly 2024-05-18 18:34:24 +00:00
Erik Schultheis
bf36a4b6b5 improved numerical error checking: 
tighter tolarances
  relative tolerance based of bf16 epsilon
  less verbose output if all is OK
 2024-05-18 13:27:35 +03:00
Andrej Karpathy
f226a2c9d9 Merge branch 'ngc92-alloc' 2024-05-17 15:13:48 +00:00
Andrej Karpathy
c8fa7a8c63 revert the adamw allocation to previous. minor 2024-05-17 15:13:36 +00:00
Erik Schultheis
b24279c4db remove duplicate workspace allocation 2024-05-17 17:37:47 +03:00
Erik Schultheis
88c3bea890 print message before actual allocation for more informative OOM behaviour 2024-05-17 17:37:45 +03:00
Erik Schultheis
0f23723ae4 joined optimizer state allocation 2024-05-17 17:37:16 +03:00
Andrej
b554d4d093
Merge pull request #423 from anthonix/readme_link 
Add link to AMD fork
 2024-05-16 21:35:18 +01:00
Anthony Blake
3113656e3f Add link to AMD fork 2024-05-16 13:33:34 -07:00
Andrej
109f516367
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs 
Free cooperative groups implementation of online softmax forward
 2024-05-16 21:23:56 +01:00
Andrej
9e04deabe3
Merge pull request #416 from azret/feature/cpu_dev 
Setting up dev/cpu area with the first matmul_forward.c
 2024-05-16 21:12:55 +01:00
Andrej Karpathy
130712215d Merge branch 'backward-bias' of https://github.com/ngc92/llm.c 2024-05-16 20:00:56 +00:00
Andrej Karpathy
68eea0bd5f Fix ZeRO-1, we were communicating way too much 2024-05-16 19:48:24 +00:00
Andrej Karpathy
6cfc7c5ce3 Merge branch 'zero-update' of https://github.com/ngc92/llm.c into ngc92-zero-update 2024-05-16 19:30:40 +00:00
Andrej
bd7dc7af96
Merge pull request #422 from karpathy/feature/recompute 
feature/recompute
 2024-05-16 20:25:31 +01:00
Andrej Karpathy
d7581fc542 make recompute be an int instead of bool, so we can strengthen it over time just like ZeRO stages, as we recompute more and more of the model in the future possibly. and make it default on because it is awesome 2024-05-16 19:07:39 +00:00
Erik Schultheis
fbd8f03eea fixup profiling 2024-05-16 14:25:06 +03:00
Erik Schultheis
8b57cf6535 reduce communication overhead for ZERO stage 1 2024-05-16 14:12:29 +03:00
Erik Schultheis
57f70ea66b simplify multi-gpu logic by reducing #ifdefs 2024-05-16 14:12:14 +03:00
Andrej Karpathy
3277ccc35d Merge branch 'act-recomp' of https://github.com/ngc92/llm.c into ngc92-act-recomp 2024-05-16 10:26:23 +00:00
Andrej
fbc5cba1f9
Merge pull request #415 from azret/feature/eot 
train_gpt.c: Removing the hardcoded GPT2_EOT
 2024-05-16 10:45:22 +01:00
Erik Schultheis
d48c3a4945 (optionally) recompute gelu activations to reduce activation memory 2024-05-16 12:39:28 +03:00
Erik Schultheis
7b810c1a3b update main training script 2024-05-16 01:04:58 +03:00
Erik Schultheis
858c6e6dae deterministic kernel 2024-05-16 01:04:10 +03:00
Erik Schultheis
2ccdfb70e0 general cleanup 2024-05-16 01:04:10 +03:00
Andrej
0f48a28f12
Merge pull request #412 from karpathy/feature/grad_accum3 
[wip] gradient accumulation, another attempt
 2024-05-15 22:53:16 +01:00
Andrej Karpathy
8eb3a432d9 revert all streams and synchronization events, we'll bring them back but only one at a time and very very carefully, because they cause really subtle bugs and issues 2024-05-15 21:09:46 +00:00
Azret Botash
160b3bd007 Setting up dev/CPU area with the first matmul_forward.c 2024-05-14 18:22:39 -07:00
Azret Botash
16f9dad301
Update train_gpt2.c 2024-05-14 17:45:08 -07:00
Azret Botash
7de50af511 train_gpt.c: Removing the hardcoded GPT2_EOT 2024-05-14 17:38:09 -07:00
Azret Botash
3c3c965840 Adding Mersenne Twisters C 2024-05-14 17:16:24 -07:00
Andrej Karpathy
a4567ae940 delete parallels, still not fixed 2024-05-14 22:13:54 +00:00
Andrej Karpathy
92fc26eba4 the nuts and bolts of gradient accumulation again, merged to master, but there is a bug and it doesn't work, debugging... 2024-05-14 21:32:47 +00:00
Erik Schultheis
b2a5508b84 constness fixes 2024-05-14 22:48:59 +03:00
Erik Schultheis
3b5933ecfb considerably speed up CPU matmul while still keeping it relatively readable 2024-05-14 22:32:24 +03:00
Andrej Karpathy
2d43e5bc97 remove legacy comment 2024-05-14 19:14:07 +00:00
Andrej
222d59fa2f
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda 
Layernorm backward updates
 2024-05-14 20:13:09 +01:00
Erik Schultheis
e553e2f084 update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results 2024-05-14 20:44:50 +03:00
Erik Schultheis
dd8c9f5ec9 fix layernorm backward: accumulate weight gradient 2024-05-14 20:43:51 +03:00
Andrej Karpathy
2346cdac93 Merge branch 'ngc92-backward-bias' 2024-05-13 20:19:09 +00:00
Andrej Karpathy
37a96debbf Merge branch 'backward-bias' of https://github.com/ngc92/llm.c into ngc92-backward-bias 2024-05-13 20:18:33 +00:00
Andrej
750c5fd7e8
Merge pull request #309 from ahrefs/zero-stage1 
Zero Redundancy Optimizer - Stage1
 2024-05-13 21:08:53 +01:00
Erik Schultheis
c66e48c06c fixup comment 2024-05-13 20:58:07 +03:00
Erik Schultheis
65727d5a4d fix CI compile by disabling kernel 5 2024-05-13 19:20:03 +03:00
Erik Schultheis
49ee3c8307 fix non-atomic version: 
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
 2024-05-13 18:27:56 +03:00
Erik Schultheis
aa41b32624 update main file 2024-05-13 17:48:25 +03:00
Erik Schultheis
081d224b21 automatically switch to buffer-less version if that can fill up the GPU 2024-05-13 17:39:32 +03:00
Erik Schultheis
c0329ebdba new kernel version with fewer atomics 2024-05-13 17:18:27 +03:00
chinthysl
f613ce895b Fix copy and cast params to master weights 2024-05-13 09:13:02 +00:00