Andrej Karpathy
da59861d38
make hellaswag optional eval yay
2024-05-22 22:42:37 +00:00
Andrej Karpathy
67239d9b8f
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
2024-05-22 22:21:53 +00:00
Andrej Karpathy
051f3ca53c
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
2024-05-22 19:24:20 +00:00
Andrej Karpathy
edb0df967a
continued changes for sharded dataloader
2024-05-22 11:59:09 +00:00
Andrej Karpathy
31310282e1
extend dataloader to be sharded
2024-05-21 21:33:24 +00:00
Andrej Karpathy
d53608820a
the write_datafile function accepts python list, which i think is faster but i didn't check
2024-05-21 17:55:56 +00:00
Andrej Karpathy
1defbd4a19
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
2024-05-21 17:05:27 +00:00
Andrej Karpathy
4bbd01fe12
add fineweb, and add the first version of a new write_shard function that contains a header properly
2024-05-21 00:16:07 +00:00
Andrej Karpathy
7d11b7996c
i also forgot to include the readme file for the new dev/data dir
2024-05-20 23:13:42 +00:00
Andrej Karpathy
71774b3c3d
oops forgot to include data_common.py
2024-05-20 23:10:49 +00:00
Andrej Karpathy
722e5b2fe5
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
2024-05-20 22:43:40 +00:00
Andrej Karpathy
c2d12f725e
small touchups to grad clip
2024-05-19 17:07:55 +00:00
Andrej Karpathy
da9f4e21b0
Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clipping
2024-05-19 16:03:53 +00:00
Andrej Karpathy
8d55c4a6ac
fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy
2024-05-19 15:45:28 +00:00
Andrej Karpathy
4e14b52285
report both acc and acc_norm
2024-05-19 15:18:04 +00:00
Andrej Karpathy
7bd2389c1f
add mmlu as well and refine both a bit
2024-05-19 14:23:51 +00:00
Andrej Karpathy
aec9ce5d0a
move hellaswag file to dev
2024-05-19 12:18:02 +00:00
Erik Schultheis
66ce5766e0
fixed up dev/cuda
2024-05-18 23:06:44 +03:00
Erik Schultheis
d7a81ef26f
added a useful mixed precision utility for dev/cuda
2024-05-18 22:45:44 +03:00
Andrej
109f516367
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs
...
Free cooperative groups implementation of online softmax forward
2024-05-16 21:23:56 +01:00
Andrej
9e04deabe3
Merge pull request #416 from azret/feature/cpu_dev
...
Setting up dev/cpu area with the first matmul_forward.c
2024-05-16 21:12:55 +01:00
Erik Schultheis
858c6e6dae
deterministic kernel
2024-05-16 01:04:10 +03:00
Erik Schultheis
2ccdfb70e0
general cleanup
2024-05-16 01:04:10 +03:00
Azret Botash
160b3bd007
Setting up dev/CPU area with the first matmul_forward.c
2024-05-14 18:22:39 -07:00
Andrej Karpathy
2d43e5bc97
remove legacy comment
2024-05-14 19:14:07 +00:00
Andrej
222d59fa2f
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda
...
Layernorm backward updates
2024-05-14 20:13:09 +01:00
Erik Schultheis
e553e2f084
update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results
2024-05-14 20:44:50 +03:00
Erik Schultheis
dd8c9f5ec9
fix layernorm backward: accumulate weight gradient
2024-05-14 20:43:51 +03:00
Erik Schultheis
c66e48c06c
fixup comment
2024-05-13 20:58:07 +03:00
Erik Schultheis
65727d5a4d
fix CI compile by disabling kernel 5
2024-05-13 19:20:03 +03:00
Erik Schultheis
49ee3c8307
fix non-atomic version:
...
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
2024-05-13 18:27:56 +03:00
Erik Schultheis
081d224b21
automatically switch to buffer-less version if that can fill up the GPU
2024-05-13 17:39:32 +03:00
Erik Schultheis
c0329ebdba
new kernel version with fewer atomics
2024-05-13 17:18:27 +03:00
Erik Schultheis
2287da0120
enable bf16
2024-05-12 19:42:32 +03:00
Erik Schultheis
ec92368f8b
added current backward bias kernel to dev cuda
2024-05-12 19:24:43 +03:00
lancer
5c90845f7a
update the -lcublas -lcublasLt flag in the comment
2024-05-09 22:04:02 -07:00
Andrej Karpathy
43c48707a7
Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2
2024-05-09 23:09:53 +00:00
ngc92
75ec629f52
remove duplicate function
2024-05-10 00:03:30 +02:00
ngc92
5920143d47
remove warning noise
2024-05-10 00:02:37 +02:00
ngc92
691c1df969
fused layernorm+residual
2024-05-09 23:48:11 +02:00
ngc92
99e765de8d
moved bf16 boilerplate to common.h
2024-05-09 11:19:24 +02:00
Joe Halabi
69aa64cc80
Adds cuDNN install instructions for a smaller but sufficient package
...
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).
- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)
- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions
Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com>
2024-05-08 14:53:31 -07:00
KarhouTam
6d7a99cb4e
Adjust comments
2024-05-08 17:45:14 +08:00
KarhouTam
6a52d8619e
Optimize codes and comments
2024-05-08 16:12:39 +08:00
KarhouTam
b3e8a9fe67
Implementation of online softmax forward kernel without cgs.
2024-05-08 09:22:14 +08:00
ademeure
e1f89b3043
remove BF16 default from classified_fused before PR
2024-05-07 00:57:01 +01:00
ademeure
1ea7f9bf25
tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changes
2024-05-07 00:54:50 +01:00
ademeure
9b55ea8cf5
More crazy optimisations to layernorm_backward, fused_classifier, and matmul_backward_bias.
2024-05-07 00:11:42 +01:00
lancer
bbfe8c989c
Minor update on the code
2024-05-05 16:59:58 -07:00
lancer
c81adeb6b2
move gelu_backward to backward block
2024-05-02 17:36:12 -07:00
