aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-28 20:35:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

1022 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ross Wheeler
c60d7de466 Adding manual trigger for CI 2024-05-27 16:50:06 -07:00
lancerts
47c670b885 amend 2024-05-27 15:13:11 -07:00
lancerts
c4c985cb87 spell explicitly uint to unsigned int 2024-05-27 14:44:47 -07:00
lancerts
f27ca4df41 fix the issue Mismatch of dweight at layernorm_backward.cu 2024-05-27 14:39:57 -07:00
Andrej Karpathy
888c0d4be3 also update the other files 2024-05-27 21:15:09 +00:00
Andrej Karpathy
eed242702a experiment with adding the llmc lib directory 2024-05-27 21:10:28 +00:00
Andrej
b67880a879
Merge pull request #471 from karpathy/feature/resume 
resume training v1
 2024-05-27 13:38:52 -07:00
Andrej Karpathy
24a8c03cb6 Merge branch 'master' into feature/resume 2024-05-27 20:30:30 +00:00
Andrej Karpathy
69d0583ac8 conditionally include dirent on not windows 2024-05-27 20:28:52 +00:00
Andrej
3b4d8bece6
Merge pull request #474 from rosslwheeler/Windows-dirent-fix 
Adding dirent.h support to unistd.h
 2024-05-27 13:27:58 -07:00
Ross Wheeler
bb80f93618 Minor cleanup 
Added static inline to the functions
 2024-05-27 13:16:34 -07:00
Ross Wheeler
758acdffbf Adding dirent.h support to unistd.h 2024-05-27 13:08:07 -07:00
Andrej Karpathy
63f0e25f5a make compiler happy 2024-05-27 19:32:31 +00:00
Andrej Karpathy
b75738c4eb resume optimization, seems to be working 2024-05-27 18:02:51 +00:00
Andrej Karpathy
f93a30fbbf more careful with conditional MPI use 2024-05-27 15:54:55 +00:00
Andrej Karpathy
d295cb8d81 part 1 of v1 of resume training functionality, writes the files but doesn't load them yet, coming up in a bit 2024-05-27 15:49:03 +00:00
Andrej
12999f7082
Merge pull request #470 from ngc92/bugfix 
bugfix for layernorm correctness (dev/cuda did not trip, but end to end test `make test_gpt2cu PRECISION=FP32 && ./test_gpt2cu
` did)
 2024-05-27 06:39:56 -07:00
Erik Schultheis
7dc3b7d7dc bugfix 2024-05-27 16:36:44 +03:00
Andrej
4d6f80e02b
Merge pull request #469 from ahrefs/zero1-torch 
Optional Zero1 changes for pytorch training
 2024-05-27 06:04:00 -07:00
Andrej
c3097ba308
Merge pull request #468 from ngc92/layernorm-bw-deadlock 
Layernorm deadlock
 2024-05-27 05:48:42 -07:00
Erik Schultheis
b913905f84 fix out-of-bounds access for encoder_backward: we were allocating too little memory (/ -> DIV_CEIL) 2024-05-27 13:15:36 +03:00
Erik Schultheis
841c4dec03 update main training file 2024-05-27 13:15:36 +03:00
Erik Schultheis
5f73ecfcf1 fix C % 256 != 0 2024-05-27 13:15:36 +03:00
Erik Schultheis
d35daf19c9 fail fast and hard; don't go into the deadlock 2024-05-27 13:15:36 +03:00
Erik Schultheis
3b6808269a some comments and optimized shared memory amount 2024-05-27 13:15:36 +03:00
Erik Schultheis
9eba86f0a5 fully vectorized smem access 2024-05-27 13:15:36 +03:00
Erik Schultheis
cb8cc25e3a utilities for Packed and more vectorization 2024-05-27 13:15:36 +03:00
Erik Schultheis
53ee3297dd use vectorized access to shared memory 2024-05-27 13:15:36 +03:00
Chinthaka Gamanayakege
f2809a6bdb zero changes for torch 2024-05-27 06:44:54 +00:00
Andrej Karpathy
ee6fa5ec9f fill in the other model sizes 2024-05-26 18:04:43 +00:00
Andrej Karpathy
5ec4b774f2 simple jupyter notebook example on how to parse and vis the logfiles 2024-05-26 17:40:49 +00:00
Andrej Karpathy
f145c1b101 fix the other files due to dataloader change 2024-05-26 17:01:32 +00:00
Andrej Karpathy
c6680a25ae fix a bug where the max_steps was not accounting for the gradient accumulation, meaning that e.g. with grad accum 2, we'd accidentally train for 2 epochs instead of just 1 epoch. also simplify the interface a bit and the code make more readable etc 2024-05-26 16:49:30 +00:00
Andrej
5d65c5f52e
Merge pull request #464 from karpathy/attempt_fix_zero1 
Attempt fix zero1
 2024-05-25 17:21:04 -07:00
Andrej Karpathy
05fb642ae1 attempt fix zero 1 2024-05-25 23:49:16 +00:00
Erik Schultheis
7ba3ef6bac make sure to run Adam only for the range of parameters that is available on the local device 2024-05-25 23:20:42 +03:00
Andrej Karpathy
d0c014372c add ability to export fineweb100B as well 2024-05-25 17:21:51 +00:00
Andrej Karpathy
f5d041ab0c careful with NULL and checkpoint correctly 2024-05-25 16:32:18 +00:00
Andrej Karpathy
273925d10d save checkpoint files to log directory, and small readability changes 2024-05-25 16:05:29 +00:00
Andrej Karpathy
90a745c40c add MFU (model flops utilization) calculation and print it. also refactor the printing a bit to make it nice. this calculation only works for A100 fp16 we should think through how to generalize it a bit 2024-05-25 14:30:59 +00:00
Andrej
ab090a80c5
Merge pull request #461 from karpathy/feature/write_checkpoints 
fix bug check null
 2024-05-25 07:01:26 -07:00
Andrej Karpathy
a13b2bd10b fix bug check null 2024-05-25 13:50:56 +00:00
Andrej
fe698b32f1
Merge pull request #457 from karpathy/feature/write_checkpoints 
add checkpoint function write to file
 2024-05-24 17:56:53 -07:00
Andrej Karpathy
2a0f78d617 attempt to fix windows episode 2 2024-05-25 00:44:16 +00:00
Andrej Karpathy
5b96d91893 attempt to fix windows? 2024-05-25 00:39:40 +00:00
Andrej Karpathy
f2ee356a06 resolve merge conflict 2024-05-25 00:33:12 +00:00
Andrej
4ff0412751
Merge pull request #450 from rosslwheeler/dataloader_win_fixes 
Add glob() for windows
 2024-05-24 17:30:35 -07:00
Andrej
6a7fd56d7c
Merge pull request #439 from lancerts/matmul-fix 
Fix the unsupported block_size in matmul_backward_bias kernel 1
 2024-05-24 17:26:37 -07:00
Andrej
5ef2c04a39
Merge pull request #458 from karpathy/feature/weight_decay_2d_only 
add weight decay
 2024-05-24 17:22:49 -07:00
Andrej Karpathy
9f08882051 add weight decay, but only for 2D tensors, as done in GPT series and in general too. this forces us to break up our adamw kernel again into one call per tensor, so there is a small throughput hit, of about 0.5% for me. but we have to break up this kernel in near future anyway 2024-05-25 00:14:10 +00:00