aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-28 20:35:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

1022 commits

This branch
This branch
All branches
Author SHA1 Message Date
“Christopher
665d0a4f89 Added example of removing cooperative groups 2024-05-02 02:09:57 +00:00
Jane Illarionova
d6d0d50627 fix typo 2024-05-02 00:18:55 +00:00
Jane Illarionova
d0e5fea2dd update residual_forward to use util functions 2024-05-02 00:17:47 +00:00
Jane Illarionova
7203c7875f remove int cast 2024-05-02 00:17:47 +00:00
Jane Illarionova
4698164e3e update train to multiply block size 2024-05-02 00:17:47 +00:00
Jane Illarionova
52b4dd02a1 update residual forward to allow kernels to use both types 2024-05-02 00:17:47 +00:00
Jane Illarionova
32df703a05 Update train_gpt2.cu 2024-05-02 00:17:47 +00:00
Jane Illarionova
4ffcf5b90a Update residual_forward.cu 2024-05-02 00:17:47 +00:00
Jane Illarionova
3de3c53ba9 update kernel with util functions 2024-05-01 23:58:49 +00:00
Jane Illarionova
d98e5ae859 remove int casting 2024-05-01 23:36:59 +00:00
Jane Illarionova
e9a80b5d84 update ceildiv for gelu_backward 2024-05-01 23:36:59 +00:00
Jane Illarionova
7746217433 update gelu backward rto allow all kernels to use both types 2024-05-01 23:36:59 +00:00
Jane Illarionova
ab2de05a13 Update train_gpt2.cu 2024-05-01 23:36:59 +00:00
Jane Illarionova
80072802d7 Create gelu_backward.cu 2024-05-01 23:36:59 +00:00
Andrej
4dd1ab43de
Merge pull request #328 from karpathy/feature/fp32_weight_master_copy 
feature/fp32 weight master copy
 2024-05-01 16:07:19 -07:00
Andrej Karpathy
c177c2694c make nice and print 2024-05-01 23:06:23 +00:00
Andrej Karpathy
795f8b690c fixes to keep master copy in fp32 of weights optionally 2024-05-01 22:50:22 +00:00
Yusheng (Ethan) Su
24abcbf6d2
Merge branch 'karpathy:master' into master 2024-05-02 02:19:16 +04:00
Yusheng Su
ac2d635007 add required package (requests) into requirements.txt 2024-05-02 02:11:38 +04:00
Andrej Karpathy
fd474feec8 Merge branch 'fp32-weights' of https://github.com/ngc92/llm.c 2024-05-01 22:02:55 +00:00
Erik Schultheis
df9f0efde6 updated letter and properly free buffer 2024-05-02 01:00:36 +03:00
Andrej
f4f7a987e7
Merge pull request #324 from eymay/init_weights_from_scratch 
Refactoring parameter size filling
 2024-05-01 14:50:39 -07:00
Erik Schultheis
e9c76a1753 option to keep weights as fp32 2024-05-02 00:11:07 +03:00
Andrej
ab95a11190
Merge pull request #325 from ngc92/dev-cuda-utils 
mixed precision utilities for dev/cuda
 2024-05-01 12:54:54 -07:00
Erik Schultheis
b42db706b2 this kernel will be released later, sorry for the spoiler :) 2024-05-01 22:47:24 +03:00
Erik Schultheis
91bc72d0c0 fixup 2024-05-01 22:40:26 +03:00
Erik Schultheis
06980b6b2f mixed precision utilities for dev/cuda 2024-05-01 22:35:55 +03:00
Andrej
d37639a8cb
Merge pull request #316 from rosslwheeler/designated_init_fix 
uint and designated initializers non-standard for C++ / Cuda
 2024-05-01 12:13:12 -07:00
Andrej
be5cee21e6
Merge pull request #317 from ChrisDryden/patch-5 
Removed detail about layernorm inputs being f32, no longer the case
 2024-05-01 12:00:01 -07:00
Andrej
38b478d8f9
Merge pull request #323 from karpathy/feature/cudnn 
feature/cudnn for flash-attention
 2024-05-01 11:34:16 -07:00
Andrej Karpathy
7d0042d574 make training data configurable and also dont move the engire dataset to device that is not a good idea 2024-05-01 18:15:10 +00:00
Andrej Karpathy
8aa9a9ed53 and also profile lol 2024-05-01 17:56:01 +00:00
Andrej Karpathy
1ed9381fd0 fix cudnnDestroy outside of ifdef 2024-05-01 17:55:36 +00:00
Andrej Karpathy
69d40ebfff add readme and search for cudd in home dir 2024-05-01 17:43:50 +00:00
Andrej Karpathy
35640651b3 updates to makefile: docs, override, etc 2024-05-01 17:32:52 +00:00
Andrej Karpathy
04718fbbd1 round 1 of small fixes for cudnn support 2024-05-01 17:07:10 +00:00
ademeure
1147983b8c Add missing cudnnDestroy() calls 2024-05-01 17:19:51 +01:00
ademeure
d9e7a0a378 Fixed profile_gpt2.cu/test_gpt2.cu for cuDNN 2024-05-01 17:03:28 +01:00
ademeure
2b540b61da attention_forward.cu ~matches train_gpt2cu & allocate less memory for stats in "att" 2024-05-01 16:53:01 +01:00
ademeure
be54842622 Rename lowp_float to floatX in /dev/cuda/attention_forward.cu 2024-05-01 15:43:28 +01:00
ademeure
c778dc0098 slightly increase wte threshold to make test_gpt2.cu pass (?) 2024-05-01 15:40:30 +01:00
ademeure
57c09529d9 Merge branch 'cudnn_try2' of https://github.com/ademeure/llm.c into cudnn_try2 2024-05-01 15:32:13 +01:00
ademeure
7beff72f5f Add makefile changes for cuDNN 2024-05-01 15:31:30 +01:00
ademeure
c4ecc04dc3 Merge remote-tracking branch 'karpathy/master' into cudnn_try2 2024-05-01 15:30:00 +01:00
ademeure
d508463ad7 Merge remote-tracking branch 'origin/cudnn_try2' into cudnn_try2 2024-05-01 15:27:07 +01:00
ademeure
d42ee0d925 Cleanup + do not allocate gradient preatt memory with cuDNN + allocate as small a cuDNN workspace as possible 2024-05-01 15:26:25 +01:00
ademeure
61ad4b15ab Fully working forward+backward cuDNN BF16 Flash Attention in train_gpt2.cu! 2024-05-01 15:02:47 +01:00
Chris Dryden
1806062a62
Removed detail about layernorm inputs being f32, no longer the case 2024-04-30 22:01:14 -07:00
Ross Wheeler
229d70b2d5 uint and designated initializers non-standard for C++ 
uint isn't defined by the MS compiler and designated initializers aren't part of C++ until C++ 20.
 2024-04-30 18:46:54 -07:00
Andrej
40da8cf233
Merge pull request #314 from jrhemstad/patch-3 
Add llm.cpp fork to README
 2024-04-30 17:03:25 -07:00