Andrej Karpathy
1f91bfc442
fix small bug on eval logging
2024-05-23 18:19:25 +00:00
Andrej Karpathy
86682af9a9
llm.c matches pytorch init from scratch exactly now
2024-05-23 17:50:33 +00:00
Andrej Karpathy
70a9c75348
use pytorch rand and fix dumb bug lol
2024-05-23 15:16:49 +00:00
Andrej Karpathy
bc1ebc110a
Merge branch 'feature/rand' of https://github.com/azret/llm.c into feature/random_init
2024-05-23 14:59:44 +00:00
Andrej Karpathy
e6a7d1d3e9
allow the python script to also init from random and save those weights, so it's a good reference for our C implementation
2024-05-23 14:59:25 +00:00
Andrej Karpathy
d09631807a
first draft of random init, crashes with some cuBLAS error, debugging
2024-05-23 14:35:46 +00:00
Andrej Karpathy
d3cf9e211c
fix bug free only if initted
2024-05-23 00:49:32 +00:00
Andrej
eda0c2f591
Merge pull request #447 from karpathy/feature/hellaswagc
...
HellaSwag eval in C
2024-05-22 15:50:12 -07:00
Andrej Karpathy
da59861d38
make hellaswag optional eval yay
2024-05-22 22:42:37 +00:00
Andrej Karpathy
67239d9b8f
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
2024-05-22 22:21:53 +00:00
Andrej Karpathy
051f3ca53c
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
2024-05-22 19:24:20 +00:00
Andrej
69f12214bf
Merge pull request #444 from karpathy/feature/sharded_data
...
extend dataloader to be sharded
2024-05-22 06:21:48 -07:00
Andrej Karpathy
099d30f814
add a super small crappy glob for windows that only matches a single unique file. this will make CI happy but we can't train on sharded data on windows until this is improved
2024-05-22 13:08:18 +00:00
Andrej Karpathy
05be4f6d82
readme changes
2024-05-22 12:32:25 +00:00
Andrej Karpathy
edb0df967a
continued changes for sharded dataloader
2024-05-22 11:59:09 +00:00
Andrej Karpathy
31310282e1
extend dataloader to be sharded
2024-05-21 21:33:24 +00:00
Andrej Karpathy
967420d1d1
fix print format warning for size_t vs int
2024-05-21 19:30:02 +00:00
Andrej Karpathy
587506d096
torch tensor can't handle uint16 so let's convert to int32, which is silly because we'll convert to .long right after but ok
2024-05-21 19:22:09 +00:00
Andrej
170c92b771
Merge pull request #440 from karpathy/feature/organize_data
...
refactor datasets
you'll have to re-generate the dataset .bin files, for example, `python dev/data/tinyshakespeare.py`.
see main README section "major changes log" for more details.
2024-05-21 11:32:38 -07:00
Andrej Karpathy
54ccbd300c
docs on master-breaking changes around dataset file representation
2024-05-21 18:31:19 +00:00
Andrej Karpathy
7d58fd2abb
adjust py file as well and make the errors better
2024-05-21 18:29:55 +00:00
Andrej Karpathy
f7cb77f3d9
docs on master-breaking change around how we store data .bin files
2024-05-21 18:20:51 +00:00
Andrej Karpathy
ccc240caab
make comment more helpful
2024-05-21 18:01:35 +00:00
Andrej Karpathy
d53608820a
the write_datafile function accepts python list, which i think is faster but i didn't check
2024-05-21 17:55:56 +00:00
Andrej Karpathy
9bad49a99a
also fix the python file. that should be it now, w.r.t. the new token format .bin files
2024-05-21 17:46:14 +00:00
Andrej Karpathy
666145e7b8
also fix the .c file and fp32 file
2024-05-21 17:30:56 +00:00
Andrej Karpathy
1defbd4a19
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
2024-05-21 17:05:27 +00:00
Andrej Karpathy
f7cbb30316
step 2 of dataloader refactor: separate out the dataloader to its own file, change its signature a little bit, and (notably) change from cudaMallocHost to a simple malloc, so that this file doesn't have to be cuda aware
2024-05-21 15:32:46 +00:00
Andrej Karpathy
4bbd01fe12
add fineweb, and add the first version of a new write_shard function that contains a header properly
2024-05-21 00:16:07 +00:00
Andrej Karpathy
c51cd70cfb
fix a slip in gitignore, i think i am getting tired today
2024-05-20 23:22:31 +00:00
Andrej Karpathy
bced34d71e
adjust gitignore
2024-05-20 23:15:44 +00:00
Andrej Karpathy
7d11b7996c
i also forgot to include the readme file for the new dev/data dir
2024-05-20 23:13:42 +00:00
Andrej Karpathy
71774b3c3d
oops forgot to include data_common.py
2024-05-20 23:10:49 +00:00
Andrej Karpathy
f671cf9288
more changes, trying to help people out because when this merges to master it will brick everyone's code...
2024-05-20 23:02:41 +00:00
Andrej Karpathy
722e5b2fe5
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
2024-05-20 22:43:40 +00:00
Andrej
6c8bc172cc
Merge pull request #436 from ChrisDryden/boundscheck
...
Moved bounds checks to outside of the kernel
2024-05-20 00:41:41 +01:00
Andrej
2751fa0880
Merge pull request #435 from ChrisDryden/warpsize
...
Added warpsize as a constant for better compile time optimization and standardization
2024-05-20 00:39:15 +01:00
Christopher
6de1137e0e
Moved bounds checks outside of kernel into assertions
2024-05-19 23:17:45 +00:00
Christopher
ead5d35973
Added warpsize as a constant for better compile time optimization and standardization
2024-05-19 22:54:47 +00:00
Andrej
1809740a1c
Merge pull request #433 from junaire/jun/some_fix
...
Check if file exists using platform specific commands & Add *.o to gi…
2024-05-19 21:02:04 +01:00
Andrej
00c291c679
Merge pull request #434 from ngc92/profile-fix
...
initialize multi_gpu_config so profile doesn't crash
2024-05-19 20:57:11 +01:00
Erik Schultheis
299ce659ec
initialize multi_gpu_config so profile doesn't crash
2024-05-19 22:25:00 +03:00
Andrej Karpathy
5a26799efb
Merge branch 'ngc92-gradient-clipping'
2024-05-19 17:51:44 +00:00
Andrej Karpathy
bc58cd1dc1
fix small bugs in grad clip, introduce a GPU CPU synch point to communicate the float grad_clip to print it, and small printing changes
2024-05-19 17:51:29 +00:00
Andrej Karpathy
c2d12f725e
small touchups to grad clip
2024-05-19 17:07:55 +00:00
Andrej Karpathy
da9f4e21b0
Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clipping
2024-05-19 16:03:53 +00:00
Jun Zhang
9e645314e6
Check if file exists using platform specific commands & Add *.o to gitignore
...
Signed-off-by: Jun Zhang <jun@junz.org>
2024-05-20 00:00:21 +08:00
Andrej
4a899f5685
Merge pull request #431 from karpathy/feature/hellaswag
...
add hellaswag and mmlu PyTorch reference eval
2024-05-19 16:53:38 +01:00
Andrej Karpathy
8d55c4a6ac
fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy
2024-05-19 15:45:28 +00:00
Andrej Karpathy
4e14b52285
report both acc and acc_norm
2024-05-19 15:18:04 +00:00
