aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-26 20:15:08 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

29 commits

Author SHA1 Message Date
lancerts
47c670b885 amend 2024-05-27 15:13:11 -07:00
lancerts
c4c985cb87 spell explicitly uint to unsigned int 2024-05-27 14:44:47 -07:00
lancerts
f27ca4df41 fix the issue Mismatch of dweight at layernorm_backward.cu 2024-05-27 14:39:57 -07:00
Erik Schultheis
7dc3b7d7dc bugfix 2024-05-27 16:36:44 +03:00
Erik Schultheis
5f73ecfcf1 fix C % 256 != 0 2024-05-27 13:15:36 +03:00
Erik Schultheis
d35daf19c9 fail fast and hard; don't go into the deadlock 2024-05-27 13:15:36 +03:00
Erik Schultheis
3b6808269a some comments and optimized shared memory amount 2024-05-27 13:15:36 +03:00
Erik Schultheis
9eba86f0a5 fully vectorized smem access 2024-05-27 13:15:36 +03:00
Erik Schultheis
cb8cc25e3a utilities for Packed and more vectorization 2024-05-27 13:15:36 +03:00
Erik Schultheis
53ee3297dd use vectorized access to shared memory 2024-05-27 13:15:36 +03:00
ademeure
7cbeefc7f3 added new layernorm backward to /dev/cuda/ 2024-05-21 23:26:54 +01:00
Erik Schultheis
e553e2f084 update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results 2024-05-14 20:44:50 +03:00
Erik Schultheis
dd8c9f5ec9 fix layernorm backward: accumulate weight gradient 2024-05-14 20:43:51 +03:00
lancer
5c90845f7a update the -lcublas -lcublasLt flag in the comment 2024-05-09 22:04:02 -07:00
ngc92
75ec629f52 remove duplicate function 2024-05-10 00:03:30 +02:00
ngc92
99e765de8d moved bf16 boilerplate to common.h 2024-05-09 11:19:24 +02:00
ademeure
9b55ea8cf5 More crazy optimisations to layernorm_backward, fused_classifier, and matmul_backward_bias. 2024-05-07 00:11:42 +01:00
Andrej Karpathy
ec73086a83 Merge branch 'removing_cg_example' of https://github.com/ChrisDryden/llm.c into ChrisDryden-removing_cg_example 2024-05-02 18:14:14 +00:00
“Christopher
665d0a4f89 Added example of removing cooperative groups 2024-05-02 02:09:57 +00:00
Erik Schultheis
06980b6b2f mixed precision utilities for dev/cuda 2024-05-01 22:35:55 +03:00
ademeure
1e50f3b976 Only enable kernel4 with BF16 2024-04-29 01:55:06 +01:00
ademeure
82fef03acd Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes) 2024-04-29 01:51:10 +01:00
austinvhuang
d937f93aaa Add a local convenience Makefile for dev/cuda/, fix 2 minor warnings observed when building all targets. 2024-04-26 21:19:44 -04:00
Andrej Karpathy
e3bcae6b2b incorporate layernorm backward kernel 2, we are now down to 73.1ms/iter on average 2024-04-22 18:44:43 +00:00
Chris Dryden
2752212151 Added shared memory for the atomic additions for the layernorm_backward.cu weights and biases 2024-04-21 16:37:08 -07:00
lancer
c327923df9 Constness fix layernorm_backward.cu 2024-04-17 14:56:03 -07:00
lancer
81aa596251 Constness fix layernorm_backward.cu 2024-04-17 14:54:49 -07:00
Erik Schultheis
c1bfe4a639 warp-wide reduction for layernorm backward 2024-04-17 21:52:40 +03:00
Andrej Karpathy
6be8080501 add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far 2024-04-16 02:10:45 +00:00