Andrej Karpathy
938f8f78bb
more defensive
2024-04-29 20:25:29 +00:00
Andrej Karpathy
b4c346ac2a
don't hardcode the EOT token and store it in the Tokenizer file
2024-04-29 20:23:41 +00:00
ademeure
c24bb88bce
Add back FP16 and disable multi_gpu all reduce when there is only 1 GPU
2024-04-29 21:15:52 +01:00
Andrej Karpathy
699c25469a
separate out our Tokenizer
2024-04-29 19:54:18 +00:00
Erik Schultheis
4872c57164
comments
2024-04-29 22:46:33 +03:00
Erik Schultheis
9f07a176e0
simplify to only support cs
2024-04-29 22:43:52 +03:00
ademeure
66c454874f
avoid warning for cublas_compute_type in BF16 mode
2024-04-29 20:25:07 +01:00
ademeure
1d4effd5a4
Remove FloatN and associated code for adam & allreduce
2024-04-29 20:21:34 +01:00
Erik Schultheis
896b644cb9
slight simplification
2024-04-29 22:19:12 +03:00
Andrej
1310e2f77d
Merge pull request #286 from rosslwheeler/Makefile-windows-support
...
Makefile windows support
2024-04-29 12:11:12 -07:00
Andrej
977c688b9d
Merge pull request #294 from karpathy/ademeure-layernormbf16-3
...
Ademeure layernormbf16 3
2024-04-29 12:10:07 -07:00
Andrej Karpathy
0e4506ac30
unify these two
2024-04-29 19:03:49 +00:00
ademeure
8be7370a55
gpt2_build_from_checkpoint
2024-04-29 19:54:32 +01:00
Erik Schultheis
45252d434c
yet another gelu
2024-04-29 21:53:08 +03:00
ademeure
ebedadfee8
Fix train_gpt2.py to write all weights as BF16 (+merge fixes)
2024-04-29 19:49:57 +01:00
ademeure
d23bb1fd13
Merge remote-tracking branch 'karpathy/master' into layernormbf16
2024-04-29 19:41:54 +01:00
Andrej
e92ec3ffd9
Merge pull request #281 from ChrisDryden/templateCleanup
...
Removed templates from kernels to clarify data types and simplify code
2024-04-29 10:05:29 -07:00
Andrej
237593f47b
Merge pull request #287 from tairov/fix-zig-link
...
Fix link to zig port in README.md
2024-04-29 09:04:18 -07:00
Ross Wheeler
fcef97d6a1
Adding back PFLAGS
...
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
2024-04-29 08:53:50 -07:00
Rickard Hallerbäck
8915d5ab0d
Adding the example kernels to CI
2024-04-29 17:02:05 +02:00
ademeure
49ffd68317
Fix sizeof(T) -> sizeof(D) (was overallocating memory)
2024-04-29 15:47:59 +01:00
Aydyn Tairov
d0b943ba04
Fix link to zig port
2024-04-29 11:18:03 +01:00
Ross Wheeler
81b689ed01
Merge branch 'Makefile-windows-support' of https://github.com/rosslwheeler/llm.c into Makefile-windows-support
2024-04-29 03:00:56 -07:00
Ross Wheeler
c9ab8ee123
Updating Makefile with latest changes
...
Add check with Windows OS
2024-04-29 03:00:46 -07:00
Ross Wheeler
4b80126913
Add check with Windows OS
2024-04-29 02:46:31 -07:00
Eymen Ünay
0e38ac7b8c
Refactoring parameter size filling
...
Similar to a new function is defined for parameter size table. This will be useful for reuse with e.g. initializing weights from scratch instead of a checkpoint.
2024-04-29 12:35:21 +03:00
Ross Wheeler
f3b1bb70df
Updating Makefile with latest changes
2024-04-29 02:28:03 -07:00
Christopher Dryden
cf3e6efc79
Changed ordering of type configuration to easily see unchanged values
2024-04-29 02:28:23 +00:00
Christopher Dryden
109c62c6c6
Removed templates from kernels to clarify data types
2024-04-29 02:06:59 +00:00
ademeure
1e50f3b976
Only enable kernel4 with BF16
2024-04-29 01:55:06 +01:00
ademeure
82fef03acd
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
2024-04-29 01:51:10 +01:00
Andrej Karpathy
50acc125f3
Merge branch 'ngc92-split-file'
...
Separates out common error-checking wrapper utils, that are broadly useful across all file
2024-04-28 23:27:08 +00:00
Andrej Karpathy
c20497cfc1
add comments pointing to the definition of the utils functions
2024-04-28 23:26:58 +00:00
Andrej Karpathy
b1c80e9d7b
Merge branch 'split-file' of https://github.com/ngc92/llm.c into ngc92-split-file
2024-04-28 23:23:00 +00:00
Erik Schultheis
4a3c278f3a
moved checked helper functions into a separate file
2024-04-29 00:00:37 +03:00
Andrej
18b41b4df6
Merge pull request #279 from Ricardicus/prec-ci
...
Updating the CI to build different precisions
2024-04-28 13:47:48 -07:00
Andrej Karpathy
66a92c6791
Merge branch 'ngc92-cleanup'
2024-04-28 20:45:04 +00:00
Andrej Karpathy
518565654e
Merge branch 'cleanup' of https://github.com/ngc92/llm.c into ngc92-cleanup
2024-04-28 20:43:37 +00:00
Andrej Karpathy
49228b091a
add small comment on -t=0
2024-04-28 20:20:51 +00:00
Erik Schultheis
4c295c7e97
and even more cleanup
2024-04-28 23:19:36 +03:00
Erik Schultheis
ca48791522
as promised, cleanup enabled by padding :)
2024-04-28 23:19:36 +03:00
Rickard Hallerbäck
b522333851
Updating the CI to build different precisions
2024-04-28 22:17:47 +02:00
Andrej
10aa24e2cc
Merge pull request #269 from ChrisDryden/patch-3
...
Enable multithreading in nvcc
2024-04-28 13:16:47 -07:00
Andrej Karpathy
4b6f68a9a9
Merge branch 'lancerts-encoder_forward-float4'
2024-04-28 20:10:48 +00:00
Andrej Karpathy
327eef3f22
incorporate faster encoder_forward kernel to fp32 CUDA version
2024-04-28 20:10:33 +00:00
Andrej Karpathy
4b6a532b9b
Merge branch 'encoder_forward-float4' of https://github.com/lancerts/llm.c into lancerts-encoder_forward-float4
2024-04-28 19:49:16 +00:00
Andrej Karpathy
b7972ff928
make padded vocab fixes in the .c code as well, i missed it in the previous PR, should satisfy the CI now
2024-04-28 19:33:21 +00:00
Andrej Karpathy
835060e18d
padded vocab change. touched a lot of code. very stressful and error prone, but i think it is done. had to bump versions on all .bin files, invalidating the previous files. re-run the python training script to re-export the new version files. let's not do too much of things like this in the future lol. actually, fun fact i had a chance to do the padded vocab really really early in the history of llm.c development, and chose not do it, thinking i'll just do it later. i should have done it. such is life, you make mistakes, you accumulate scar tissue, and you learn, and you become better, faster, stronger. this is the mindset one must have to lead a happy and fulfilling life. it's not important that you are perfect at any point in time, it's only important that you keep improving, every day.
2024-04-28 18:47:03 +00:00
Andrej
d95b8d8320
Merge pull request #265 from karpathy/feature/load_bf16
...
load bf16 directly, and some "quality of life" handling of fp32/fp16/bf16 precisions
2024-04-28 09:24:56 -07:00
Andrej Karpathy
9d70d9a1b2
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
2024-04-28 16:17:22 +00:00
