mirror of https://github.com/karpathy/llm.c.git synced 2026-07-27 20:25:09 -04:00

History

Andrej Karpathy 31be5e790d add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool		2024-09-26 03:17:58 +00:00
..
README.md	add RoPE PyTorch and C reference code	2024-09-24 23:52:16 +00:00
rmsnorm.py	add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool	2024-09-26 03:17:58 +00:00
rope.c	add RoPE PyTorch and C reference code	2024-09-24 23:52:16 +00:00
rope.py	add RoPE PyTorch and C reference code	2024-09-24 23:52:16 +00:00

README.md

cbridge

We'll use this directory for the PyTorch -> C bridge. So we have some PyTorch code and we'd like to have the equivalent C implementation (usually that one in turn serves as reference for the CUDA kernels later).

For starters we have RoPE. E.g. generate the reference with PyTorch and then match it in C:

python rope.py
gcc -o rope rope.c -lm
./rope

The .py file writes a robe.bin file with the intermediate tensors.