aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-23 19:45:08 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

242 commits

Author SHA1 Message Date
Andrej
80e0c3b8d9
Merge pull request #512 from gordicaleksa/refactor_encoder_bwd_kernel 
Remove redundant CPU computation in encoder bwd
 2024-06-02 16:43:10 -07:00
Aleksa Gordic
290c00a362 Remove redundant CPU computation 2024-06-01 18:27:24 +02:00
Andrej
a26041af12
Merge pull request #495 from ChrisDryden/shared_memory 
Removed unnecesary shared memory due to blockreduce using static defined shared memory
 2024-06-01 08:12:21 -07:00
Aleksa Gordic
8eafd40615 Update compile cmd in the dev/cuda README 2024-06-01 16:46:32 +02:00
Christopher
5450632237 Removed unnecesary shared memory due to blockreduce using static defined shared memory 2024-05-30 03:01:52 +00:00
Andrej
55f3665b64
Merge pull request #493 from vyom1611/master-2 
Modal benchmarking script updated to replace deprecated calls
 2024-05-29 07:50:10 -07:00
vyom1611
062f096da1 Modal benchmarking script updated to replace deprecated calls 2024-05-29 20:01:41 +05:30
lancerts
47c670b885 amend 2024-05-27 15:13:11 -07:00
lancerts
c4c985cb87 spell explicitly uint to unsigned int 2024-05-27 14:44:47 -07:00
lancerts
f27ca4df41 fix the issue Mismatch of dweight at layernorm_backward.cu 2024-05-27 14:39:57 -07:00
Erik Schultheis
7dc3b7d7dc bugfix 2024-05-27 16:36:44 +03:00
Erik Schultheis
5f73ecfcf1 fix C % 256 != 0 2024-05-27 13:15:36 +03:00
Erik Schultheis
d35daf19c9 fail fast and hard; don't go into the deadlock 2024-05-27 13:15:36 +03:00
Erik Schultheis
3b6808269a some comments and optimized shared memory amount 2024-05-27 13:15:36 +03:00
Erik Schultheis
9eba86f0a5 fully vectorized smem access 2024-05-27 13:15:36 +03:00
Erik Schultheis
cb8cc25e3a utilities for Packed and more vectorization 2024-05-27 13:15:36 +03:00
Erik Schultheis
53ee3297dd use vectorized access to shared memory 2024-05-27 13:15:36 +03:00
Andrej
6a7fd56d7c
Merge pull request #439 from lancerts/matmul-fix 
Fix the unsupported block_size in matmul_backward_bias kernel 1
 2024-05-24 17:26:37 -07:00
Andrej Karpathy
dbacaf84cf Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm 2024-05-24 21:46:25 +00:00
Erik Schultheis
1b98637960 int -> int64_t 2024-05-24 20:11:34 +03:00
ademeure
7cbeefc7f3 added new layernorm backward to /dev/cuda/ 2024-05-21 23:26:54 +01:00
lancer
2b0667aee1 update the utils function and assert 2024-05-20 08:00:39 -07:00
lancer
6348d4196d fix the unsupported block_size 2024-05-19 17:39:25 -07:00
Andrej Karpathy
c2d12f725e small touchups to grad clip 2024-05-19 17:07:55 +00:00
Erik Schultheis
66ce5766e0 fixed up dev/cuda 2024-05-18 23:06:44 +03:00
Erik Schultheis
d7a81ef26f added a useful mixed precision utility for dev/cuda 2024-05-18 22:45:44 +03:00
Andrej
109f516367
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs 
Free cooperative groups implementation of online softmax forward
 2024-05-16 21:23:56 +01:00
Erik Schultheis
858c6e6dae deterministic kernel 2024-05-16 01:04:10 +03:00
Erik Schultheis
2ccdfb70e0 general cleanup 2024-05-16 01:04:10 +03:00
Andrej Karpathy
2d43e5bc97 remove legacy comment 2024-05-14 19:14:07 +00:00
Andrej
222d59fa2f
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda 
Layernorm backward updates
 2024-05-14 20:13:09 +01:00
Erik Schultheis
e553e2f084 update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results 2024-05-14 20:44:50 +03:00
Erik Schultheis
dd8c9f5ec9 fix layernorm backward: accumulate weight gradient 2024-05-14 20:43:51 +03:00
Erik Schultheis
c66e48c06c fixup comment 2024-05-13 20:58:07 +03:00
Erik Schultheis
65727d5a4d fix CI compile by disabling kernel 5 2024-05-13 19:20:03 +03:00
Erik Schultheis
49ee3c8307 fix non-atomic version: 
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
 2024-05-13 18:27:56 +03:00
Erik Schultheis
081d224b21 automatically switch to buffer-less version if that can fill up the GPU 2024-05-13 17:39:32 +03:00
Erik Schultheis
c0329ebdba new kernel version with fewer atomics 2024-05-13 17:18:27 +03:00
Erik Schultheis
2287da0120 enable bf16 2024-05-12 19:42:32 +03:00
Erik Schultheis
ec92368f8b added current backward bias kernel to dev cuda 2024-05-12 19:24:43 +03:00
lancer
5c90845f7a update the -lcublas -lcublasLt flag in the comment 2024-05-09 22:04:02 -07:00
Andrej Karpathy
43c48707a7 Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2 2024-05-09 23:09:53 +00:00
ngc92
75ec629f52 remove duplicate function 2024-05-10 00:03:30 +02:00
ngc92
5920143d47 remove warning noise 2024-05-10 00:02:37 +02:00
ngc92
691c1df969 fused layernorm+residual 2024-05-09 23:48:11 +02:00
ngc92
99e765de8d moved bf16 boilerplate to common.h 2024-05-09 11:19:24 +02:00
Joe Halabi
69aa64cc80 Adds cuDNN install instructions for a smaller but sufficient package 
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).

- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)

- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions

Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com>
 2024-05-08 14:53:31 -07:00
KarhouTam
6d7a99cb4e Adjust comments 2024-05-08 17:45:14 +08:00
KarhouTam
6a52d8619e Optimize codes and comments 2024-05-08 16:12:39 +08:00
KarhouTam
b3e8a9fe67 Implementation of online softmax forward kernel without cgs. 2024-05-08 09:22:14 +08:00