mirror of
https://github.com/karpathy/llm.c.git
synced 2026-07-28 20:35:09 -04:00
add comment documenting how to reproduce python reference exactly
This commit is contained in:
parent
050cbfa42c
commit
70b4de8700
1 changed files with 7 additions and 0 deletions
|
|
@ -24,6 +24,13 @@ Also we're using TinyStories here for example as it is a bigger dataset
|
|||
|
||||
Example launch using bfloat16 on 4 GPUs, same as above:
|
||||
mpirun -np 4 ./train_gpt2cu -b 8 -v 200 -s 200 -i data/TinyStories
|
||||
|
||||
If you'd like to see train_gpt2.cu produce identical results to
|
||||
`python train_gpt2.py`, you can run it like this:
|
||||
make train_gpt2cu PRECISION=FP32
|
||||
./train_gpt2cu -b 4 -t 64 -l 1e-4 -v 200 -s 200 -a 1 -x 10 -f 0
|
||||
This reads & runs in fp32, B=4, T=64, LR=1e-4, val/sample never (200),
|
||||
-a 1 is "overfit single batch", -x 10 is 10 iterations, and -f 0 disables tf32
|
||||
*/
|
||||
|
||||
#include <stdio.h>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue