aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-26 20:15:08 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

make comment on comparison to PyTorch runtime with kernels as they are right now

Browse source
This commit is contained in:
Andrej Karpathy 2024-04-18 04:53:09 +00:00
parent 959ea543dc
commit eef92c3fbe
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -149,7 +149,7 @@ And being gone to find some gentleman to his place
Zilled me with his eagle; and thus to find my light
```
So as currently configured, we come down to validation loss of 3.51 and can generate okay samples. This runs on my A100 in about ~30 seconds.
So as currently configured, we come down to validation loss of 3.51 and can generate okay samples. This runs on my A100 in about ~30 seconds. Even so, this training loop in PyTorch is about 80ms/iteration, so we are currently ~5X slower. The kernels are being actively optimized and worked on and we aspire to reach the same vicinity soon^TM.
**Timing**. Finally we can time the code and compare the speed to PyTorch. Because the backward pass CUDA kernels are still very new and being optimized, I'd recommend turning off the backward + update in `train_gpt2.cu`: