aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-27 20:25:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1605 commits 52 branches 0 tags 21 MiB
Commit graph

388 commits

Author SHA1 Message Date
Andrej Karpathy
7d945e994c reshuffle repkv a bit, i wrote it from scratch. the kernel is still correct. repkv backward looks correct. rope backward is trivial so i don't see how it's not correct, and i also checked it. basically i'm really confused right now 2024-09-27 19:25:09 +00:00
Andrej Karpathy
8d49062293 add backward kernel to dev/cuda for rope, to ensure correctness. but i mean, it's trivial. this can't possibly be the issue. it must be the repkv 2024-09-27 18:07:10 +00:00
Andrej Karpathy
28e4a7f83e small fixes, but still not too happy with this kernel, it wastes thread and more efficient implementation kernel2 is desireable and desired 2024-09-27 01:13:32 +00:00
Insop Song
a2b66f153a Merge remote-tracking branch 'upstream/llama3' into insop/llama3 2024-09-26 11:42:31 -07:00
Andrej Karpathy
31be5e790d add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool 2024-09-26 03:17:58 +00:00
Insop Song
d1f2f64541 Updated repkv_backward cuda kernel 
- kernel 1 is tested

- build
```
make repkv_backward
/usr/local/cuda/bin/nvcc -O3 --use_fast_math --generate-code arch=compute_80,code=[compute_80,sm_80] -lcublas -lcublasLt -std=c++17 repkv_backward.cu -o repkv_backward
```

- test run on A30
```
Using kernel 1
Checking block size 32.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 64.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 128.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 256.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 512.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 1024.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
All results match. Starting benchmarks.

block_size   32 time 3.2461 ms
block_size   64 time 1.7509 ms
block_size  128 time 1.7374 ms
block_size  256 time 1.7441 ms
block_size  512 time 1.8092 ms
block_size 1024 time 2.0443 ms
```
 2024-09-25 17:40:38 -07:00
Insop Song
3e5134d94f Merge branch 'insop/llama3_wip' into insop/llama3 2024-09-25 15:53:17 -07:00
Insop Song
8874c2c3f6 Merge remote-tracking branch 'upstream/llama3' into insop/llama3 2024-09-25 15:50:51 -07:00
Andrej Karpathy
52c7254267 add the finished RoPE forward pass 2024-09-25 19:00:27 +00:00
Insop Song
8336d2a055 Merge remote-tracking branch 'upstream/llama3' into insop/llama3 2024-09-24 18:31:03 -07:00
Andrej Karpathy
026e4ed323 add RoPE PyTorch and C reference code 2024-09-24 23:52:16 +00:00
Insop Song
de9c8170e5 wip 2024-09-23 20:52:50 -07:00
Insop Song
fcc3466b2f Add minor change 2024-09-22 09:05:56 -07:00
Insop Song
42d09e8732 minor clean up 2024-09-22 00:26:29 -07:00
Insop Song
ad46043aaf WIP updating cuda kernel 2024-09-22 00:22:26 -07:00
Insop Song
6c68657c6b Put cuda kernel caller placeholder 2024-09-21 21:30:45 -07:00
Insop Song
080e57fd46 CPU version tested 
- [ ] WIP cuda version
 2024-09-21 21:21:13 -07:00
Insop Song
45401b42eb DRAFT: Adding backward kernel for repkv 
- [ ] WIP: CPU kernel
- [ ] Cuda kernel
 2024-09-21 20:40:57 -07:00
Andrej Karpathy
56f956cc4a add repkv kernel to replicate K,V heads after the QKV projection 2024-09-21 01:53:08 +00:00
Andrej Karpathy
755458d0e1 fix tokenizer omg 2024-08-13 17:24:15 +00:00
Andrej Karpathy
16635d41a2 attempt to fix PR 2024-08-13 02:13:14 +00:00
Aleksa Gordic
68624cc6ac Refactor tokenize, delegate instead 2024-08-12 20:26:21 +02:00
Aleksa Gordic
e8739c89a0 Refactor magic/version header dict 2024-08-12 20:17:53 +02:00
Aleksa Gordic
eb1b359dae Replace fire with argparse 2024-08-10 22:45:46 +02:00
Aleksa Gordic
50765638c0 Add fineweb llama 3 support 2024-08-10 22:35:46 +02:00
Aleksa Gordic
f8c1ff590f Use fire module, update readme 2024-08-10 22:29:19 +02:00
Aleksa Gordic
15b76bf0fb Add tiny shakespeare 2024-08-10 22:23:33 +02:00
Aleksa Gordic
71419cbd61 Update header description 2024-08-10 21:47:09 +02:00
Aleksa Gordic
d3151c4ace Add llama 3 tokenization and data loader 2024-08-10 21:25:57 +02:00
Andrej Karpathy
1566d6b9f6 revert makefile change 2024-07-30 20:07:54 +00:00
mspronesti
574ef9b87d chore(permute): use common.h utils for validation and benchmarking 2024-07-29 00:19:31 +01:00
mspronesti
a49da7d917 make: generalize TARGETS definition 2024-07-29 00:19:03 +01:00
Andrej
b4623bc5e7
Merge pull request #712 from indianspeedster/dev-cuda 
Added permute kernel in dev/cuda
 2024-07-28 06:08:20 -10:00
mspronesti
c6e74ea322 fix(dev/cuda): memory leaks 2024-07-26 00:33:51 +01:00
indianspeedster
a80bce65e5 Added compilation steps in makefile for permute.cu 2024-07-25 18:02:06 -04:00
indianspeedster
8952c524af added kernel to permute in dev/cuda 2024-07-25 17:35:29 -04:00
Ross Wheeler
9dc920a66f Move WIN32 code into unistd.h 2024-07-20 12:05:20 -07:00
Ross Wheeler
56621bba0e Adding CI check for exceeding loss tolerance 
LLMC-26
 2024-07-13 01:14:20 -07:00
Andrej Karpathy
48fee049cb add edu fineweb sh download script, change paths to my HF repo, add hellaswag to starter pack 2024-07-01 21:41:15 +00:00
Andrej Karpathy
ea36d04e39 small cosmetic fixes to fineweb data file 2024-06-30 15:52:51 +00:00
Andrej
2daaf6ab83
Merge pull request #629 from rosslwheeler/feature/tests 
CI Dataloader test and ptx/sass file generator
 2024-06-29 15:17:31 -07:00
Andrej Karpathy
ff7a90bd8d Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_support 2024-06-28 19:35:44 +00:00
Ross Wheeler
d6d08be3c0 Addressed review comments 
Explicit precision set for tests
 2024-06-27 00:58:19 -07:00
rosslwheeler
267efc9a0e
Merge branch 'karpathy:master' into feature/tests 2024-06-26 22:38:21 -07:00
Andrej Karpathy
e12d090483 resolve merge conflicts 2024-06-26 04:05:46 +00:00
Ross Wheeler
2e6fd9b150 Add unistd.h for Windows cuDNN build 
Small fix - move WIN32 change into dev.unistd.h

Add USE_CUDNN=1 build for Windows for CI
 2024-06-25 10:36:59 -07:00
rosslwheeler
2e32e25e37
Merge branch 'karpathy:master' into feature/tests 2024-06-24 23:05:55 -07:00
Andrej Karpathy
569f9f16f9 add outlier detector, test for it, and start tracking z score of the loss, for now only print it 2024-06-25 01:37:31 +00:00
Andrej Karpathy
2543b62b88 Merge branch 'streams-io' of https://github.com/ngc92/llm.c into ngc92-streams-io 2024-06-23 00:48:05 +00:00
Ross Wheeler
5b95bff118 Dataloader test and ptx/sass file generator 
Setup new test infrastructure
 2024-06-21 17:23:03 -07:00