Andrej Karpathy
|
2a0f78d617
|
attempt to fix windows episode 2
|
2024-05-25 00:44:16 +00:00 |
|
Andrej Karpathy
|
5b96d91893
|
attempt to fix windows?
|
2024-05-25 00:39:40 +00:00 |
|
Andrej
|
4ff0412751
|
Merge pull request #450 from rosslwheeler/dataloader_win_fixes
Add glob() for windows
|
2024-05-24 17:30:35 -07:00 |
|
Andrej
|
6a7fd56d7c
|
Merge pull request #439 from lancerts/matmul-fix
Fix the unsupported block_size in matmul_backward_bias kernel 1
|
2024-05-24 17:26:37 -07:00 |
|
Ross Wheeler
|
79738d2ca4
|
fixed mkdir change
|
2024-05-24 17:05:59 -07:00 |
|
Ross Wheeler
|
e5083be900
|
Moved windows glob() over to dev/unistd.h
Added header guard and changed long->int64_t in dataloader.h
|
2024-05-24 16:56:52 -07:00 |
|
Andrej Karpathy
|
dbacaf84cf
|
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
|
2024-05-24 21:46:25 +00:00 |
|
Andrej
|
bf03e7fd4c
|
Merge pull request #456 from ngc92/large-batch-fix
fix for large batch sizes
|
2024-05-24 12:00:41 -07:00 |
|
Erik Schultheis
|
1b98637960
|
int -> int64_t
|
2024-05-24 20:11:34 +03:00 |
|
Andrej Karpathy
|
08fe3eef43
|
change default shard size for fineweb10B
|
2024-05-24 15:56:05 +00:00 |
|
Erik Schultheis
|
b66eb66ac7
|
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 17:45:26 +03:00 |
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
ademeure
|
7cbeefc7f3
|
added new layernorm backward to /dev/cuda/
|
2024-05-21 23:26:54 +01:00 |
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
Andrej Karpathy
|
d53608820a
|
the write_datafile function accepts python list, which i think is faster but i didn't check
|
2024-05-21 17:55:56 +00:00 |
|
Andrej Karpathy
|
1defbd4a19
|
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
|
2024-05-21 17:05:27 +00:00 |
|
Andrej Karpathy
|
4bbd01fe12
|
add fineweb, and add the first version of a new write_shard function that contains a header properly
|
2024-05-21 00:16:07 +00:00 |
|
Andrej Karpathy
|
7d11b7996c
|
i also forgot to include the readme file for the new dev/data dir
|
2024-05-20 23:13:42 +00:00 |
|
Andrej Karpathy
|
71774b3c3d
|
oops forgot to include data_common.py
|
2024-05-20 23:10:49 +00:00 |
|
Andrej Karpathy
|
722e5b2fe5
|
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
|
2024-05-20 22:43:40 +00:00 |
|
lancer
|
2b0667aee1
|
update the utils function and assert
|
2024-05-20 08:00:39 -07:00 |
|
lancer
|
6348d4196d
|
fix the unsupported block_size
|
2024-05-19 17:39:25 -07:00 |
|
Andrej Karpathy
|
c2d12f725e
|
small touchups to grad clip
|
2024-05-19 17:07:55 +00:00 |
|
Andrej Karpathy
|
da9f4e21b0
|
Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clipping
|
2024-05-19 16:03:53 +00:00 |
|
Andrej Karpathy
|
8d55c4a6ac
|
fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy
|
2024-05-19 15:45:28 +00:00 |
|
Andrej Karpathy
|
4e14b52285
|
report both acc and acc_norm
|
2024-05-19 15:18:04 +00:00 |
|
Andrej Karpathy
|
7bd2389c1f
|
add mmlu as well and refine both a bit
|
2024-05-19 14:23:51 +00:00 |
|
Andrej Karpathy
|
aec9ce5d0a
|
move hellaswag file to dev
|
2024-05-19 12:18:02 +00:00 |
|
Erik Schultheis
|
66ce5766e0
|
fixed up dev/cuda
|
2024-05-18 23:06:44 +03:00 |
|
Erik Schultheis
|
d7a81ef26f
|
added a useful mixed precision utility for dev/cuda
|
2024-05-18 22:45:44 +03:00 |
|
Andrej
|
109f516367
|
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs
Free cooperative groups implementation of online softmax forward
|
2024-05-16 21:23:56 +01:00 |
|
Andrej
|
9e04deabe3
|
Merge pull request #416 from azret/feature/cpu_dev
Setting up dev/cpu area with the first matmul_forward.c
|
2024-05-16 21:12:55 +01:00 |
|
Erik Schultheis
|
858c6e6dae
|
deterministic kernel
|
2024-05-16 01:04:10 +03:00 |
|
Erik Schultheis
|
2ccdfb70e0
|
general cleanup
|
2024-05-16 01:04:10 +03:00 |
|
Azret Botash
|
160b3bd007
|
Setting up dev/CPU area with the first matmul_forward.c
|
2024-05-14 18:22:39 -07:00 |
|
Andrej Karpathy
|
2d43e5bc97
|
remove legacy comment
|
2024-05-14 19:14:07 +00:00 |
|
Andrej
|
222d59fa2f
|
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda
Layernorm backward updates
|
2024-05-14 20:13:09 +01:00 |
|
Erik Schultheis
|
e553e2f084
|
update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results
|
2024-05-14 20:44:50 +03:00 |
|
Erik Schultheis
|
dd8c9f5ec9
|
fix layernorm backward: accumulate weight gradient
|
2024-05-14 20:43:51 +03:00 |
|
Erik Schultheis
|
c66e48c06c
|
fixup comment
|
2024-05-13 20:58:07 +03:00 |
|
Erik Schultheis
|
65727d5a4d
|
fix CI compile by disabling kernel 5
|
2024-05-13 19:20:03 +03:00 |
|
Erik Schultheis
|
49ee3c8307
|
fix non-atomic version:
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
|
2024-05-13 18:27:56 +03:00 |
|
Erik Schultheis
|
081d224b21
|
automatically switch to buffer-less version if that can fill up the GPU
|
2024-05-13 17:39:32 +03:00 |
|
Erik Schultheis
|
c0329ebdba
|
new kernel version with fewer atomics
|
2024-05-13 17:18:27 +03:00 |
|
Erik Schultheis
|
2287da0120
|
enable bf16
|
2024-05-12 19:42:32 +03:00 |
|
Erik Schultheis
|
ec92368f8b
|
added current backward bias kernel to dev cuda
|
2024-05-12 19:24:43 +03:00 |
|
lancer
|
5c90845f7a
|
update the -lcublas -lcublasLt flag in the comment
|
2024-05-09 22:04:02 -07:00 |
|