aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-28 20:35:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

1022 commits

This branch
This branch
All branches
Author SHA1 Message Date
chinthysl
086ce2ff4d Remove unsused template specializations and refactor 2024-05-13 06:48:22 +00:00
chinthysl
b3a5d1da15 shard master_weights 2024-05-13 05:55:46 +00:00
Chinthaka Gamanayake
0d9ce74b74
Merge branch 'karpathy:master' into zero-stage1 2024-05-13 13:34:52 +08:00
Erik Schultheis
2287da0120 enable bf16 2024-05-12 19:42:32 +03:00
Andrej
c1814d5d2f
Merge pull request #401 from rosslwheeler/ci-add-win 
Add Windows to CI
 2024-05-12 18:32:13 +02:00
Andrej
a2bffb5a8a
Merge pull request #403 from nietras/nietras-llm-cs 
Add Llm.cs by nietras to README
 2024-05-12 18:29:20 +02:00
Erik Schultheis
ec92368f8b added current backward bias kernel to dev cuda 2024-05-12 19:24:43 +03:00
ntr
e64df91191 Add Llm.cs by nietras to README 2024-05-12 16:02:36 +02:00
Ross Wheeler
b88f683569 Add Windows to CI 
Change notes:
•	Add Windows build/test to CI matrix build
•	Add Cuda Windows build
•	Replace v3 checkout actions to v4 per GitHub recommendations
 2024-05-11 17:49:22 -07:00
Andrej Karpathy
1cfe899121 Merge branch 'master' of github.com:karpathy/llm.c 2024-05-11 17:21:18 +00:00
Andrej Karpathy
fb86c121a1 Merge branch 'Anerudhan-feature/simplify_cudnn_usage' 2024-05-11 17:21:04 +00:00
Andrej Karpathy
3dbb0bb89c bump the threshold for qkvw because flashattention expands the error here a tiny bit 2024-05-11 17:19:41 +00:00
Andrej Karpathy
58d9dda497 Merge branch 'feature/simplify_cudnn_usage' of https://github.com/Anerudhan/llm.c into Anerudhan-feature/simplify_cudnn_usage 2024-05-11 17:08:31 +00:00
Andrej
5dbda7e115
Merge pull request #397 from lancerts/comment-fix 
Update the -lcublas -lcublasLt flag in the comment
 2024-05-11 19:03:40 +02:00
Andrej
923b2b0a8f
Merge pull request #398 from zwets/fix-cudnn-frontend-check 
Fix detection of cudnn-frontend in '.' on Linux
 2024-05-10 12:13:53 +02:00
Marco van Zwetselaar
6da5e63e2c Fix detection of cudnn-frontend in '.' on Linux 
Plus minor legibility fixes in the cudnn-frontend explanation.
 2024-05-10 10:57:39 +03:00
lancer
5c90845f7a update the -lcublas -lcublasLt flag in the comment 2024-05-09 22:04:02 -07:00
Andrej
fd007d6508
Merge pull request #395 from yijunyu/patch-1 
Update README.md
 2024-05-10 01:18:25 +02:00
Andrej Karpathy
83ad9ac87f Fused Residual+LayerNorm layers, ~1% faster 2024-05-09 23:16:41 +00:00
Andrej Karpathy
43c48707a7 Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2 2024-05-09 23:09:53 +00:00
Yijun Yu
8ccf2f97f8
Update README.md 2024-05-10 06:49:24 +08:00
ngc92
75ec629f52 remove duplicate function 2024-05-10 00:03:30 +02:00
ngc92
5920143d47 remove warning noise 2024-05-10 00:02:37 +02:00
ngc92
691c1df969 fused layernorm+residual 2024-05-09 23:48:11 +02:00
Anerudhan
01c7a331bb - Simplify graph cache and usage of cudnn. 
- Fix failures in H100
 2024-05-09 19:07:00 +00:00
Andrej
077b649ee1
Merge pull request #389 from vedaanta/cudnn/install 
Adds cuDNN install instructions for a smaller but sufficient package
 2024-05-09 20:30:48 +02:00
ngc92
99e765de8d moved bf16 boilerplate to common.h 2024-05-09 11:19:24 +02:00
Joe Halabi
69aa64cc80 Adds cuDNN install instructions for a smaller but sufficient package 
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).

- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)

- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions

Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com>
 2024-05-08 14:53:31 -07:00
Andrej
b8eaafdf9b
Merge pull request #386 from Anerudhan/feature/rename_cudnn_attn 
Rename cudnn_att.cu to cudnn_att.cpp to speed up compilation.
 2024-05-08 23:00:57 +02:00
Andrej Karpathy
68509c89a2 Merge branch 'pjj-fix-gpt2-xl' 2024-05-08 20:20:27 +00:00
Andrej Karpathy
0c73ba6ca5 Merge branch 'fix-gpt2-xl' of https://github.com/pjj/llm.c into pjj-fix-gpt2-xl 2024-05-08 20:19:15 +00:00
Andrej Karpathy
26dbbc75c2 unbreak the tests oops 2024-05-08 20:18:33 +00:00
Andrej Karpathy
84a2aa0dd0 allow exporting all GPT-2 model sizes now, using --model flag in python script. also allow loading all the models from C, using the new flag -e, to point directly to the file to load. added some error handling for potentially common mistakes. note that loadin GPT-2 XL does not work yet and crashes, but I am imminently merging a fix as the next commit that resolves this 2024-05-08 20:13:14 +00:00
Anerudhan
0ce5fcf7e8 Rename cudnn_att.cu to cudnn_att.cpp to speed up compilation. 2024-05-08 19:53:48 +00:00
Paul Maragakis
99e5c5d283 Allow the code to train gpt2-xl 
This commit is the fix by adameure described in the comment below:

https://github.com/karpathy/llm.c/pull/382#issuecomment-2100895501

To reproduce the following bug:

https://github.com/karpathy/llm.c/pull/382#issuecomment-2100648148

First switch to gpt2-xl for tinystories with the following three commands:

python prepro_tinystories.py  # download tinystories

sed -i 's/from_pretrained("gpt2")/from_pretrained("gpt2-xl")/' train_gpt2.py  # select gpt2-xl

python train_gpt2.py --input_bin data/TinyStories_train.bin --batch_size 1  # Prep the large model binaries

Now you can compile and try the code with or without the fix.  On an H100 machine I use the following command:

train_gpt2cu -i data/TinyStories  -b 10 -x 100  # gets 230k/s on one H100 node of 8 GPU
 2024-05-08 15:17:25 -04:00
Andrej
1c5ef8cea3
Merge pull request #382 from ahrefs/fix-multigpu-hang 
Fix incorrect GPU assignment in multi gpu setup
 2024-05-08 19:50:00 +02:00
KarhouTam
6d7a99cb4e Adjust comments 2024-05-08 17:45:14 +08:00
KarhouTam
6a52d8619e Optimize codes and comments 2024-05-08 16:12:39 +08:00
chinthysl
2356be7333 set stream to main_stream in ncclAllReduce 2024-05-08 05:02:23 +00:00
chinthysl
dc901d420b set correct gpu using multigpu config 2024-05-08 04:50:24 +00:00
KarhouTam
b3e8a9fe67 Implementation of online softmax forward kernel without cgs. 2024-05-08 09:22:14 +08:00
Andrej Karpathy
2f6c545131 Merge branch 'ademeure-more_kernel_opt' 2024-05-07 09:36:37 +00:00
Andrej Karpathy
5b070903ce Merge branch 'more_kernel_opt' of https://github.com/ademeure/llm.c into ademeure-more_kernel_opt 2024-05-07 09:31:12 +00:00
ademeure
c261240852 3 x 512 threads max for layernorm_backward to avoid cache thrashing (hacky -> better way?) 2024-05-07 03:43:27 +01:00
Andrej
0141408ca5
Merge pull request #357 from rosslwheeler/cudnn_makefile_changes 
cuDNN makefile changes
 2024-05-07 02:50:11 +02:00
Andrej
ce72242c46
Merge pull request #376 from rosslwheeler/train_gp2cu-porting-fixes 
Replace non-standard C syntax with standard C syntax - fix win compilation
 2024-05-07 02:47:47 +02:00
Ross Wheeler
3a6f89cebc Merge branch 'train_gp2cu-porting-fixes' of https://github.com/rosslwheeler/llm.c into train_gp2cu-porting-fixes 2024-05-06 17:41:47 -07:00
Ross Wheeler
2550754215 Change FLT_MAX to flt_max 2024-05-06 17:41:19 -07:00
Andrej Karpathy
3cc16f1354 fix logits bug 2024-05-07 00:39:25 +00:00
rosslwheeler
60b819630d
Merge branch 'karpathy:master' into train_gp2cu-porting-fixes 2024-05-06 17:25:59 -07:00