aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-28 20:35:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

1022 commits

This branch
This branch
All branches
Author SHA1 Message Date
Jake Hemstad
78eba5fff3
Add llm.cpp fork to README 2024-04-30 18:52:42 -05:00
Andrej Karpathy
769c911ab6 small tweaks 2024-04-30 23:48:06 +00:00
Andrej
bad76f408a
Merge pull request #313 from ngc92/small-improvements 
fixed potential error and generalized gelu forward
 2024-04-30 16:18:08 -07:00
harry
74d5bcd03a Adding a java port of the project 2024-04-30 14:51:06 -07:00
Erik Schultheis
603d862830 fix up bandwidth calculation 2024-04-30 23:52:08 +03:00
Erik Schultheis
c3515cfe47 fixed potential error and generalized gelu forward 2024-04-30 23:45:34 +03:00
Andrej Karpathy
7ba1ed8a1a delete packed128 .fp32, use (float) ok 2024-04-30 19:39:20 +00:00
Andrej Karpathy
7fd0659ba8 Pack gelu_forward kernel using the new Packed128 2024-04-30 19:35:23 +00:00
Andrej Karpathy
9141e04693 fixes to the PR, careful with float/floatX etc 2024-04-30 19:35:05 +00:00
Andrej Karpathy
242981eecf Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing 2024-04-30 19:06:39 +00:00
Andrej Karpathy
70b4de8700 add comment documenting how to reproduce python reference exactly 2024-04-30 18:51:45 +00:00
Andrej Karpathy
050cbfa42c override train split to val split if we are debugging and trying to overfit a single batch of data, following the python script reference behavior 2024-04-30 18:48:12 +00:00
Andrej Karpathy
44656c3850 allow fp32 precision in the test script as well 2024-04-30 18:38:38 +00:00
Andrej Karpathy
b84571f745 also add argparse to force tf32 to zero 2024-04-30 17:54:15 +00:00
Andrej Karpathy
654d6f55c5 add arg to cap the number of steps, and offset all prints to start steps at 1, which i think looks better to the eye 2024-04-30 17:46:36 +00:00
Andrej Karpathy
52e2ca8378 Merge branch 'master' of github.com:karpathy/llm.c 2024-04-30 16:36:00 +00:00
Andrej
179b159a1d
Merge pull request #311 from leloykun/fc--modal-script 
Add script to run benchmarks on Modal
 2024-04-30 09:35:51 -07:00
Franz Louis Cesista
b239b67ae1 add modal script 2024-05-01 00:15:08 +08:00
Andrej Karpathy
9d8a6d13f0 add flag overfit_single_batch useful debugging into train_gpt2.cu 2024-04-30 16:03:33 +00:00
ademeure
435ac928b4 cuDNN Forward Flash Attention is working in both /dev/cuda/ and train_gpt2.cu! (backwards pass is broken for now as a conquence) 2024-04-30 17:00:09 +01:00
Andrej Karpathy
51face88d9 fix bug where backward/step must be outside of amp context 2024-04-30 15:48:40 +00:00
Andrej
7e8f14d2c0
Merge pull request #310 from ngc92/flush-l2-for-benchmark 
clear the L2 cache between benchmarks
 2024-04-30 07:23:41 -07:00
ademeure
1c516c732d re-commit of my old cudnn forward attention changes 2024-04-30 14:54:35 +01:00
Erik Schultheis
d7813d281d clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results 2024-04-30 12:45:37 +03:00
chinthysl
1f442fdf7f fix gpt2_update call 2024-04-30 09:10:50 +00:00
chinthysl
b3e8abdd54 optimizer update per shard and nccl all gather 2024-04-30 09:09:52 +00:00
chinthysl
2d26ec10b2 setting the zero opt configs 2024-04-30 09:08:25 +00:00
chinthysl
3254a51522 Zero Optimizations configs 2024-04-30 09:07:01 +00:00
Christopher Dryden
568615fef1 Added packing for gelu forwards kernel 2024-04-30 03:21:42 +00:00
Andrej
9464f4272e
Merge pull request #283 from ChrisDryden/configurationRewrite 
Changed ordering of type configuration to easily see unchanged values
 2024-04-29 18:58:03 -07:00
Andrej
997864920b
Merge pull request #290 from Ricardicus/ci-dev-cuda 
Adding the example kernels to CI
 2024-04-29 18:53:03 -07:00
Andrej
2490f7896d
Merge pull request #298 from karpathy/feature/packed128 
Feature/packed128
 2024-04-29 16:31:02 -07:00
Andrej Karpathy
906d22f5b6 fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags 2024-04-29 23:27:55 +00:00
Andrej Karpathy
af2bc47d22 remove forceinline. blame @arund42 if this ever does not get inlined by the compiler 2024-04-29 23:10:51 +00:00
Andrej Karpathy
f02ba4b436 import updated tokenizer properly 2024-04-29 23:02:26 +00:00
Andrej Karpathy
7634f0863c add packed128 to the mainline file 2024-04-29 22:57:59 +00:00
Andrej Karpathy
5e12c66f5a move packed128 to common 2024-04-29 22:55:28 +00:00
Andrej Karpathy
f43592954f reset back to precious hyperparameters 2024-04-29 22:44:25 +00:00
Andrej Karpathy
ee125cffc4 make it imo clearer that the grid size is one quarter as large as it would otherwise be. because each thread now does 4X work 2024-04-29 22:40:26 +00:00
Andrej Karpathy
083b210dd4 small changes 2024-04-29 22:38:03 +00:00
Andrej Karpathy
d9506a7a0a minor changes to packed128 2024-04-29 22:31:19 +00:00
Andrej Karpathy
7eefefb511 Merge branch 'yet-another-gelu' of https://github.com/ngc92/llm.c into ngc92-yet-another-gelu 2024-04-29 22:19:35 +00:00
Andrej Karpathy
39ce9be516 fix missing \n on a print 2024-04-29 21:05:44 +00:00
Andrej Karpathy
f66825faba allow using flash attention via argparse and print tok/s 2024-04-29 21:05:30 +00:00
Rickard Hallerbäck
5ad0079fca Reverting changes to Makefile, adding -j4 to make 2024-04-29 22:44:21 +02:00
Andrej
0d6e096b9b
Merge pull request #296 from karpathy/separate_tokenizer 
separate out our Tokenizer
 2024-04-29 13:44:05 -07:00
Andrej
1443edc746
Merge branch 'master' into separate_tokenizer 2024-04-29 13:38:58 -07:00
Andrej Karpathy
70f777617d Single call to NCCL AllReduce and AdamW for bf16 2024-04-29 20:36:29 +00:00
Andrej Karpathy
dd60fc3991 fix prints 2024-04-29 20:36:07 +00:00
Andrej Karpathy
be4aeb3d2d fix accumulated loss 2024-04-29 20:32:10 +00:00