aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-27 20:25:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
647 commits 52 branches 0 tags 21 MiB
Commit graph

193 commits

Author SHA1 Message Date
Andrej Karpathy
3048a3a780 oops have to use labels= 2024-05-06 18:04:35 +00:00
Andrej Karpathy
b4278f094a v1 of what was supposed to be a repro of wikitext ppl, but isn't, what 2024-05-06 18:03:10 +00:00
lancer
c81adeb6b2 move gelu_backward to backward block 2024-05-02 17:36:12 -07:00
Andrej Karpathy
937fbd8e36 small doc fixes 2024-05-02 22:17:22 +00:00
Erik Schultheis
6e48501d60 bias backward kernel that will use all available threads 2024-05-03 01:05:47 +03:00
Andrej Karpathy
50714d2254 pragma unroll quick fix 2024-05-02 21:06:11 +00:00
Andrej Karpathy
9da2729c5a update encoder_forward with the latest and greatest Packed128 and incorporate into train_gpt2cu 2024-05-02 20:40:05 +00:00
Andrej Karpathy
3d1761b370 Merge branch 'patch-1' of https://github.com/JaneIllario/llm.c into JaneIllario-patch-1 2024-05-02 19:43:14 +00:00
Andrej Karpathy
99f51bae1c Merge branch 'gelu_backwards' of https://github.com/JaneIllario/llm.c into JaneIllario-gelu_backwards-3 2024-05-02 19:15:06 +00:00
Andrej Karpathy
ec73086a83 Merge branch 'removing_cg_example' of https://github.com/ChrisDryden/llm.c into ChrisDryden-removing_cg_example 2024-05-02 18:14:14 +00:00
Andrej Karpathy
2202c9a51b add kernel 4 to docs. have to improve these docs more and document them better 2024-05-02 18:09:27 +00:00
“Christopher
398cdaf4cd Removed cooperative groups and added packed128 to fused classifier and prepare softmax blockwide 2024-05-02 02:33:51 +00:00
“Christopher
665d0a4f89 Added example of removing cooperative groups 2024-05-02 02:09:57 +00:00
Jane Illarionova
d6d0d50627 fix typo 2024-05-02 00:18:55 +00:00
Jane Illarionova
d0e5fea2dd update residual_forward to use util functions 2024-05-02 00:17:47 +00:00
Jane Illarionova
52b4dd02a1 update residual forward to allow kernels to use both types 2024-05-02 00:17:47 +00:00
Jane Illarionova
4ffcf5b90a Update residual_forward.cu 2024-05-02 00:17:47 +00:00
Jane Illarionova
3de3c53ba9 update kernel with util functions 2024-05-01 23:58:49 +00:00
Jane Illarionova
7746217433 update gelu backward rto allow all kernels to use both types 2024-05-01 23:36:59 +00:00
Jane Illarionova
80072802d7 Create gelu_backward.cu 2024-05-01 23:36:59 +00:00
Erik Schultheis
b42db706b2 this kernel will be released later, sorry for the spoiler :) 2024-05-01 22:47:24 +03:00
Erik Schultheis
91bc72d0c0 fixup 2024-05-01 22:40:26 +03:00
Erik Schultheis
06980b6b2f mixed precision utilities for dev/cuda 2024-05-01 22:35:55 +03:00
ademeure
1147983b8c Add missing cudnnDestroy() calls 2024-05-01 17:19:51 +01:00
ademeure
2b540b61da attention_forward.cu ~matches train_gpt2cu & allocate less memory for stats in "att" 2024-05-01 16:53:01 +01:00
ademeure
be54842622 Rename lowp_float to floatX in /dev/cuda/attention_forward.cu 2024-05-01 15:43:28 +01:00
ademeure
c4ecc04dc3 Merge remote-tracking branch 'karpathy/master' into cudnn_try2 2024-05-01 15:30:00 +01:00
Andrej Karpathy
769c911ab6 small tweaks 2024-04-30 23:48:06 +00:00
Erik Schultheis
603d862830 fix up bandwidth calculation 2024-04-30 23:52:08 +03:00
Erik Schultheis
c3515cfe47 fixed potential error and generalized gelu forward 2024-04-30 23:45:34 +03:00
Andrej Karpathy
7ba1ed8a1a delete packed128 .fp32, use (float) ok 2024-04-30 19:39:20 +00:00
Andrej Karpathy
9141e04693 fixes to the PR, careful with float/floatX etc 2024-04-30 19:35:05 +00:00
Andrej Karpathy
242981eecf Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing 2024-04-30 19:06:39 +00:00
Franz Louis Cesista
b239b67ae1 add modal script 2024-05-01 00:15:08 +08:00
ademeure
1c516c732d re-commit of my old cudnn forward attention changes 2024-04-30 14:54:35 +01:00
Erik Schultheis
d7813d281d clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results 2024-04-30 12:45:37 +03:00
Christopher Dryden
568615fef1 Added packing for gelu forwards kernel 2024-04-30 03:21:42 +00:00
Andrej Karpathy
af2bc47d22 remove forceinline. blame @arund42 if this ever does not get inlined by the compiler 2024-04-29 23:10:51 +00:00
Andrej Karpathy
5e12c66f5a move packed128 to common 2024-04-29 22:55:28 +00:00
Andrej Karpathy
f43592954f reset back to precious hyperparameters 2024-04-29 22:44:25 +00:00
Andrej Karpathy
ee125cffc4 make it imo clearer that the grid size is one quarter as large as it would otherwise be. because each thread now does 4X work 2024-04-29 22:40:26 +00:00
Andrej Karpathy
083b210dd4 small changes 2024-04-29 22:38:03 +00:00
Andrej Karpathy
d9506a7a0a minor changes to packed128 2024-04-29 22:31:19 +00:00
Andrej Karpathy
7eefefb511 Merge branch 'yet-another-gelu' of https://github.com/ngc92/llm.c into ngc92-yet-another-gelu 2024-04-29 22:19:35 +00:00
Erik Schultheis
4872c57164 comments 2024-04-29 22:46:33 +03:00
Erik Schultheis
9f07a176e0 simplify to only support cs 2024-04-29 22:43:52 +03:00
Erik Schultheis
896b644cb9 slight simplification 2024-04-29 22:19:12 +03:00
Erik Schultheis
45252d434c yet another gelu 2024-04-29 21:53:08 +03:00
ademeure
d23bb1fd13 Merge remote-tracking branch 'karpathy/master' into layernormbf16 2024-04-29 19:41:54 +01:00
ademeure
49ffd68317 Fix sizeof(T) -> sizeof(D) (was overallocating memory) 2024-04-29 15:47:59 +01:00