aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-26 20:15:08 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

279 commits

Author SHA1 Message Date
ademeure
3ba2cdc723 fixes + bounds checking 2024-04-18 03:41:27 +01:00
lancer
ac055f730c fix the bug that yields cpu, gpu results mismatch 2024-04-17 17:47:11 -07:00
Erik Schultheis
58a7b57f96 enable benchmarking 2024-04-18 01:39:47 +03:00
Erik Schultheis
e278112c19 coalesced reading in attention backward pass 2024-04-18 01:39:43 +03:00
ademeure
4ee98f5135 Merge branch 'master' into optimised_fused_classifier 2024-04-17 22:58:51 +01:00
lancer
c327923df9 Constness fix layernorm_backward.cu 2024-04-17 14:56:03 -07:00
lancer
81aa596251 Constness fix layernorm_backward.cu 2024-04-17 14:54:49 -07:00
Erik Schultheis
c1bfe4a639 warp-wide reduction for layernorm backward 2024-04-17 21:52:40 +03:00
Andrej Karpathy
fa4ebfa663 kernel2 for softmax backward 2024-04-17 08:14:22 +00:00
Andrej Karpathy
d74d7ae652 small adamw fixes, needs more work 2024-04-17 04:58:16 +00:00
Andrej Karpathy
66a0b5242e Merge branch 'master' of github.com:karpathy/llm.c 2024-04-17 04:51:26 +00:00
Andrej Karpathy
8a673edcc3 push my attention_backward file 2024-04-17 04:42:42 +00:00
nopperl
97e61e2258 add slightly more optimized adamw kernel 2024-04-17 03:31:07 +02:00
nopperl
5684ff7e77 add adamw cuda kernel 2024-04-17 03:30:59 +02:00
Erik Schultheis
43bf9eb66a ensure valid test cases and incorporate fixes by @ademeure 2024-04-16 20:53:50 +03:00
ademeure
b225501b7b optimised version of the cool new fused classifier + hopefully bugfixes? 2024-04-16 14:54:23 +01:00
lancer
6ff4ad646d fix the backward dbias precision issue 2024-04-15 19:58:16 -07:00
Andrej Karpathy
6be8080501 add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far 2024-04-16 02:10:45 +00:00
Andrej Karpathy
3de507739c minor fixes to matmul backward 2024-04-16 01:23:41 +00:00
Andrej Karpathy
9560b1a92b add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at first 2024-04-16 00:40:11 +00:00
Andrej
3c6dafeb53
Merge pull request #117 from ngc92/fused-classifier 
WIP: Fully fused classification layer
 2024-04-15 12:13:48 -07:00
Erik Schultheis
c73fa8842a fixed numerics for online kernel, and an existing race condition in kernel 2 2024-04-15 03:41:09 +03:00
Erik Schultheis
8fb966bc31 kernel 1 should also accumulate in double to ensure stability 2024-04-15 02:52:38 +03:00
Erik Schultheis
60cf281e46 use V dimension in softmax tests 
adjusted tests to be more challenging
 2024-04-15 02:14:48 +03:00
Erik Schultheis
99b42c1a94 nicer printing 2024-04-15 02:03:48 +03:00
Erik Schultheis
fe1d8cfb47 fix timing output 2024-04-15 02:03:42 +03:00
Erik Schultheis
780d8bd012 initial version 2024-04-14 03:48:56 +03:00
Andrej Karpathy
534b93a732 erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7 2024-04-13 22:02:31 +00:00
Andrej Karpathy
c5ee371564 make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common 2024-04-13 17:56:18 +00:00
Andrej Karpathy
82623518a4 fix comment block of gelu forward kernel 2024-04-13 17:48:53 +00:00
Andrej Karpathy
02b5b33c5f remove unused variable C 2024-04-13 17:48:11 +00:00
Andrej Karpathy
84145411d3 fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans 2024-04-13 17:47:19 +00:00
Andrej Karpathy
69f0385237 add comments and handling of kernels and their correctness 2024-04-13 17:44:31 +00:00
Erik Schultheis
f459bfa833 fixed re-initialization of device memory after test 2024-04-13 18:23:12 +03:00
Erik Schultheis
d035725d22 mark problematic tests 2024-04-13 18:09:28 +03:00
Erik Schultheis
59bada94d9 improved testing 2024-04-13 17:06:52 +03:00
Erik Schultheis
7e7460cfbd const'ness fixes 2024-04-13 13:38:11 +03:00
Erik Schultheis
7e8e038de8 extracted common testing functions into their own header 2024-04-13 13:00:05 +03:00
Andrej Karpathy
6b49ed1c0b incorporate the fastest kernels and adjust readme 2024-04-13 02:58:23 +00:00
Andrej Karpathy
6f9e28a1b6 few comments 2024-04-13 02:38:57 +00:00
Andrej Karpathy
2f99b3e608 Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2 2024-04-13 02:34:06 +00:00
Andrej Karpathy
40b78be722 small changes to comments, and very minor code modifications and movements 2024-04-13 02:11:36 +00:00
Andrej Karpathy
6250336c09 Merge branch 'online_softmax' of https://github.com/lancerts/llm.c into lancerts-online_softmax 2024-04-13 01:54:16 +00:00
Andrej Karpathy
a18e517c1a alright let's start the backward pass. crossentropy_softmax_backward first, ez one 2024-04-13 01:52:55 +00:00
Andrej Karpathy
59b9e8b96a fix handles in cublas in attention forward as well 2024-04-12 23:15:47 +00:00
Andrej
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it (#98) 
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
 2024-04-12 15:32:07 -07:00
Erik Schultheis
e048825f5a div_ceil 2024-04-13 01:20:09 +03:00
Erik Schultheis
8e87aac341 online softmax 2024-04-12 12:47:48 +03:00
Erik Schultheis
de98a9a44a vector loads 2024-04-12 12:47:43 +03:00
Erik Schultheis
baba3cfa01 recalculate instead of memory round-trip 2024-04-12 02:20:20 +03:00