Andrej Karpathy
888c0d4be3
also update the other files
2024-05-27 21:15:09 +00:00
Andrej Karpathy
f145c1b101
fix the other files due to dataloader change
2024-05-26 17:01:32 +00:00
Andrej Karpathy
3221e4b2d2
small cosmetic changes
2024-05-24 23:10:17 +00:00
Andrej Karpathy
6e4296fb9a
Merge branch 'cpu-matmul' of https://github.com/ngc92/llm.c into ngc92-cpu-matmul
2024-05-24 22:59:24 +00:00
Andrej Karpathy
967420d1d1
fix print format warning for size_t vs int
2024-05-21 19:30:02 +00:00
Andrej Karpathy
666145e7b8
also fix the .c file and fp32 file
2024-05-21 17:30:56 +00:00
Andrej Karpathy
f671cf9288
more changes, trying to help people out because when this merges to master it will brick everyone's code...
2024-05-20 23:02:41 +00:00
Andrej Karpathy
722e5b2fe5
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
2024-05-20 22:43:40 +00:00
Azret Botash
16f9dad301
Update train_gpt2.c
2024-05-14 17:45:08 -07:00
Azret Botash
7de50af511
train_gpt.c: Removing the hardcoded GPT2_EOT
2024-05-14 17:38:09 -07:00
Erik Schultheis
b2a5508b84
constness fixes
2024-05-14 22:48:59 +03:00
Erik Schultheis
3b5933ecfb
considerably speed up CPU matmul while still keeping it relatively readable
2024-05-14 22:32:24 +03:00
Andrej
f4f7a987e7
Merge pull request #324 from eymay/init_weights_from_scratch
...
Refactoring parameter size filling
2024-05-01 14:50:39 -07:00
Andrej Karpathy
f02ba4b436
import updated tokenizer properly
2024-04-29 23:02:26 +00:00
Eymen Ünay
0e38ac7b8c
Refactoring parameter size filling
...
Similar to a new function is defined for parameter size table. This will be useful for reuse with e.g. initializing weights from scratch instead of a checkpoint.
2024-04-29 12:35:21 +03:00
Andrej Karpathy
c20497cfc1
add comments pointing to the definition of the utils functions
2024-04-28 23:26:58 +00:00
Erik Schultheis
4a3c278f3a
moved checked helper functions into a separate file
2024-04-29 00:00:37 +03:00
Andrej Karpathy
b7972ff928
make padded vocab fixes in the .c code as well, i missed it in the previous PR, should satisfy the CI now
2024-04-28 19:33:21 +00:00
Andrej Karpathy
9c2209254b
also improve situation in the c code
2024-04-26 22:27:22 +00:00
Coenraad
2efac6670a
Make msvc compile by adding the ifdefs back in gelu_backward, fixes #200
2024-04-22 23:17:06 +02:00
Andrej
734f65f33e
Merge pull request #192 from rosslwheeler/const_fixes
...
const changes to help with Microsoft compiler OpenMP 5.2 support
2024-04-21 10:42:15 -07:00
Andrej Karpathy
adac21e97b
add a small comment as to what this is
2024-04-20 19:45:34 +00:00
Coenraad
f534e4bdfe
Fixes -Ofast optimizations breaking model by skipping them for gelu_backward
2024-04-20 21:33:00 +02:00
Ross Wheeler
e564222113
const changes to support OpenMP 5.2 MSVC work-around
...
const changes to support OpenMP MSVC work-around
2024-04-20 02:07:12 -07:00
Andrej Karpathy
5bdbc45806
only do 5 validation batches by default, we really don't need much precision here for this little script with current defaults
2024-04-20 00:45:13 +00:00
Andrej Karpathy
27ccf0b595
fix bugs, must use size_t instead of int, this was bad. and also init cpu_losses to NULL if we want to free it later
2024-04-15 22:20:44 +00:00
Coenraad
7064e36472
more original typos fixed
2024-04-15 20:22:33 +02:00
Andrej Karpathy
bcb5228ab9
also update the .c file in addition to .cu with recent changes, e.g. the tokenizer and the B*T sampling, and update the README to match
2024-04-15 02:32:26 +00:00
Andrej Karpathy
3652832073
update the .c code, not just .cu code. oh copy paste, starting to bite us haha. also fflush when printing tokens
2024-04-15 02:13:49 +00:00
Andrej Karpathy
dec54904b4
one more simple clarifying comment
2024-04-11 22:24:38 +00:00
Andrej Karpathy
d680fbad25
one more comment on attention
2024-04-11 21:49:14 +00:00
Andrej Karpathy
9710163a60
did one pass of additional comments, potentially helpful, as i think a lot of people may be looking at this for the first time
2024-04-11 21:45:46 +00:00
Andrej Karpathy
1751af1607
inline docs for sizes
2024-04-11 16:04:14 +00:00
Andrej Karpathy
f26cf00a61
make gelu constant be a #define, speeds up the kernel by 1% or so
2024-04-11 15:49:01 +00:00
Andrej Karpathy
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
karpathy
3bcb9ba7d2
remove unused variables
2024-04-08 15:10:51 -07:00
Rickard Hallerbäck
b2228d0e83
Minor correction for openmp check
2024-04-08 22:33:47 +02:00
karpathy
e8e1628632
first commit of just the reference cpu fp32 gpt2 training
2024-04-08 12:41:21 -07:00
