aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-28 20:35:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1356 commits 52 branches 0 tags 21 MiB
Commit graph

1356 commits

This branch
This branch
All branches
Author SHA1 Message Date
Andrej Karpathy
7b92930021 move things around a bit, a few comments, and also update python script to correspond to C more 2024-07-01 17:20:46 +00:00
Andrej Karpathy
8253823432 Merge branch 'fix_periodic_spikes' of https://github.com/gordicaleksa/llm.c into gordicaleksa-fix_periodic_spikes-2 2024-07-01 17:14:22 +00:00
Andrej
942fed59fe
Merge pull request #662 from karpathy/overfit-batch-actually 
fix the overfit single batch behavior to actually overfit batch, not …
 2024-07-01 10:12:20 -07:00
Andrej Karpathy
7fd7717816 fix the overfit single batch behavior to actually overfit batch, not microbatch, we do this more cleanly by simply resetting the dataloader every step 2024-07-01 17:00:40 +00:00
Aleksa Gordic
74e5ba7260 Add micro step guard 2024-07-01 12:11:02 +02:00
Aleksa Gordic
78e7627859 Remove local args for testing 2024-07-01 11:31:55 +02:00
Aleksa Gordic
de8454d2f4 Make the grad cleaning more robust 2024-07-01 11:30:36 +02:00
Aleksa Gordic
d2f06aed01 Fix periodic spikes - clean loss after val 2024-07-01 11:24:16 +02:00
Andrej Karpathy
61625d168c only rank 0 creates log dir 2024-07-01 06:31:37 +00:00
Andrej Karpathy
ea36d04e39 small cosmetic fixes to fineweb data file 2024-06-30 15:52:51 +00:00
Andrej
2daaf6ab83
Merge pull request #629 from rosslwheeler/feature/tests 
CI Dataloader test and ptx/sass file generator
 2024-06-29 15:17:31 -07:00
Andrej Karpathy
e6e365a1a8 Merge branch 'eliebak-add_edu_fineweb_support' 2024-06-28 19:35:58 +00:00
Andrej Karpathy
ff7a90bd8d Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_support 2024-06-28 19:35:44 +00:00
Andrej Karpathy
2cf7c5b7d3 Merge branch 'ngc92-mixed-dtypes-2' 2024-06-27 17:47:22 +00:00
Andrej Karpathy
012915c06c handle unknown datatype to squash compiler warning 2024-06-27 17:46:46 +00:00
Andrej
06cd83eacc
Merge pull request #652 from ademeure/cudnn_determinism 
Make cuDNN deterministic for Flash Attention backward
 2024-06-27 10:22:04 -07:00
ademeure
29a067f916 Make cuDNN deterministic for Flash Attention backward (requires frontend >=1.5) 2024-06-27 13:33:02 +00:00
Ross Wheeler
d6d08be3c0 Addressed review comments 
Explicit precision set for tests
 2024-06-27 00:58:19 -07:00
Ross Wheeler
eb0b74e559 Add H100 and A100 ptx/sass files 2024-06-27 00:30:38 -07:00
rosslwheeler
267efc9a0e
Merge branch 'karpathy:master' into feature/tests 2024-06-26 22:38:21 -07:00
ngc92
98875dd022 macro-based alternative 2024-06-27 02:31:02 +02:00
ngc92
d38c614eb9 add some comments 2024-06-26 23:58:31 +03:00
Erik Schultheis
7beaa1ed96 keep layernorm stats in fp32 2024-06-26 20:48:11 +03:00
Erik Schultheis
f0c3e35de8 switch losses to fp32 2024-06-26 20:48:08 +03:00
Erik Schultheis
ca47165d29 allocation function for mixed dtype activation 2024-06-26 20:48:03 +03:00
Andrej
68535cb6ce
Merge pull request #649 from karpathy/feature/openmpi_flexible_path 
tune openmpi make a bit more convenient
 2024-06-26 09:11:22 -07:00
Andrej Karpathy
7c283df9c4 use ?= for OPENMPI_DIR so it can be overriden 2024-06-26 16:06:54 +00:00
Andrej
c1ac121c77
Merge pull request #646 from Ricardicus/makefile-fix 
Makefile: Respect the NO_MULTI_GPU
 2024-06-26 08:42:59 -07:00
Andrej
a08e32e474
Merge pull request #648 from krrishnarraj/mention_fork 
[readme] mention opencl fork
 2024-06-26 08:41:15 -07:00
Andrej Karpathy
2a8f82f14f tune openmpi make a bit more convenient 2024-06-26 15:39:10 +00:00
Krishnaraj Bhat
0aa0eb09a6 [readme] mention opencl fork 2024-06-26 20:32:21 +05:30
Rickard Hallerbäck
66cd8c3f1f Makefile: respect the NO_MULTI_GPU 2024-06-26 14:55:02 +02:00
Andrej
44fde98344
Merge pull request #637 from karpathy/feature/outliers 
add outlier detector, test for it, and start tracking z score of loss
 2024-06-25 21:16:29 -07:00
Andrej Karpathy
e12d090483 resolve merge conflicts 2024-06-26 04:05:46 +00:00
Andrej Karpathy
ac018c3302 Merge branch 'ngc92-more-streams' 2024-06-25 23:41:55 +00:00
Andrej Karpathy
9c7f1f9701 minor comments 2024-06-25 23:41:40 +00:00
Andrej Karpathy
4f84e3daa7 Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams 2024-06-25 23:25:57 +00:00
Erik Schultheis
7171f78f04 fix check call 2024-06-26 02:21:44 +03:00
Andrej Karpathy
c0e84eb859 Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams 2024-06-25 23:16:50 +00:00
Erik Schultheis
4aae1e529b fix up profile 2024-06-26 02:11:55 +03:00
Erik Schultheis
6c24f13b55 check token validity 2024-06-26 02:01:24 +03:00
Erik Schultheis
7b13f20343 moved targets to backward/validate 2024-06-26 01:56:23 +03:00
Andrej Karpathy
496d8bf449 Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams 2024-06-25 22:44:00 +00:00
Erik Schultheis
04e1c183ec set mean_loss to -1 if not calculated 2024-06-26 01:18:19 +03:00
Andrej
16b5bd53fb
Merge pull request #636 from karpathy/feature/rolling_checkpoints 
rolling checkpoints
 2024-06-25 14:43:21 -07:00
Andrej Karpathy
11e2fd1cf2 separate out write_checkpoint and delete_checkpoint 2024-06-25 20:46:03 +00:00
Erik Schultheis
7018280448 fix zero-1 2024-06-25 21:28:52 +03:00
Erik Schultheis
09503dea4f make windows happy 2024-06-25 21:28:52 +03:00
Erik Schultheis
2413b0a6d0 fix profile 2024-06-25 21:28:52 +03:00
Erik Schultheis
39f976f013 added missing free 2024-06-25 21:28:52 +03:00