Erik Schultheis
7e8e038de8
extracted common testing functions into their own header
2024-04-13 13:00:05 +03:00
Andrej Karpathy
6b49ed1c0b
incorporate the fastest kernels and adjust readme
2024-04-13 02:58:23 +00:00
Andrej Karpathy
6f9e28a1b6
few comments
2024-04-13 02:38:57 +00:00
Andrej Karpathy
2f99b3e608
Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2
2024-04-13 02:34:06 +00:00
Andrej Karpathy
40b78be722
small changes to comments, and very minor code modifications and movements
2024-04-13 02:11:36 +00:00
Andrej Karpathy
6250336c09
Merge branch 'online_softmax' of https://github.com/lancerts/llm.c into lancerts-online_softmax
2024-04-13 01:54:16 +00:00
Andrej Karpathy
a18e517c1a
alright let's start the backward pass. crossentropy_softmax_backward first, ez one
2024-04-13 01:52:55 +00:00
Andrej Karpathy
59b9e8b96a
fix handles in cublas in attention forward as well
2024-04-12 23:15:47 +00:00
Andrej
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
Erik Schultheis
e048825f5a
div_ceil
2024-04-13 01:20:09 +03:00
Erik Schultheis
8e87aac341
online softmax
2024-04-12 12:47:48 +03:00
Erik Schultheis
de98a9a44a
vector loads
2024-04-12 12:47:43 +03:00
Erik Schultheis
baba3cfa01
recalculate instead of memory round-trip
2024-04-12 02:20:20 +03:00
lancer
69d0e5bac7
Include a parallel kernal of softmax_online
2024-04-11 15:30:36 -07:00
lancer
ca1e5ee04d
Include a parallel kernal of softmax_online
2024-04-11 15:24:41 -07:00
Erik Schultheis
5f99fb30d2
cooperative groups and fused scale kernel
2024-04-11 23:06:46 +03:00
lancer
d6bcc129b9
Improve the efficiency
2024-04-11 12:16:20 -07:00
lancer
00010c7188
Improve the efficiency
2024-04-11 12:13:34 -07:00
Andrej Karpathy
00b9fda4a1
ceildiv in layernorm
2024-04-11 18:21:34 +00:00
Andrej Karpathy
089cd5f1dd
make sure we also test correctness of mean and rstd and do it all block sizes
2024-04-11 18:12:26 +00:00
Andrej Karpathy
f28613e0b9
small edits to comments and variables
2024-04-11 18:09:54 +00:00
Andrej Karpathy
2553518c97
Merge branch 'layer-norm' of https://github.com/ngc92/llm.c into ngc92-layer-norm
2024-04-11 18:07:00 +00:00
Erik Schultheis
4e91362fa2
optionally generate mean and rstd
2024-04-11 20:48:14 +03:00
lancer
f524a90de3
Include the online softmax CPU code (from the paper "Online normalizer calculation for softmax") and native port to GPU kernel.
...
Both the CPU and GPU kernels' correctness are verified by executing the script.
Performance: the online kernel (native CPU port) is faster than the kernel1 (native CPU port) but worse than kernel 2-4.
2024-04-11 09:47:50 -07:00
Andrej Karpathy
f26cf00a61
make gelu constant be a #define, speeds up the kernel by 1% or so
2024-04-11 15:49:01 +00:00
Erik Schultheis
afddb1af5b
fixed block size and improved load op hint
2024-04-11 03:47:38 +03:00
Erik Schultheis
466723bb06
fully fused layer-norm kernel
2024-04-11 02:01:41 +03:00
lancer
68d2ce0601
fix error in small typos in matmul_forward.cu
2024-04-10 13:35:20 -07:00
Andrej
9f5069d3b2
Merge pull request #38 from lancerts/fix_matmul
...
fix the consistency of the transpose notation in matmul_foward.cu
2024-04-10 12:15:53 -07:00
Andrej Karpathy
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
Andrej Karpathy
d8e2a36f15
ok crossentropy is fairly ez, just have to be careful with logs of negative numbers in debug tests. so this should be all the layers now
2024-04-10 01:13:20 +00:00
Andrej Karpathy
46ee0a3944
sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired
2024-04-10 00:55:14 +00:00
Andrej Karpathy
919ae1f27b
ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now
2024-04-10 00:50:30 +00:00
lancer
fbd679adec
fix the consistency of the transpose notation
2024-04-09 15:19:55 -07:00
Andrej Karpathy
8386e5393c
intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice
2024-04-09 20:08:11 +00:00
Andrej Karpathy
c29d70a9b3
add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions
2024-04-09 19:22:47 +00:00
Andrej
ea1fad0337
Merge pull request #33 from lancerts/fix_M_PI_undefined
...
fix a potential error: identifier M_PI is undefined in the gelu kernal
2024-04-09 11:25:23 -07:00
Andrej Karpathy
bb4f04bdf8
add flash attention, but a naive implementation there of
2024-04-09 18:22:49 +00:00
lancer
3a4d783633
amend
2024-04-09 09:59:03 -07:00
lancer
73c92912b0
fix error: identifier M_PI is undefined
2024-04-09 09:51:28 -07:00
Andrej Karpathy
03f37cfb6b
naive attention kernel only parallelizing over batch,time,heads. have to speed this up a lot
2024-04-09 14:53:18 +00:00
Andrej Karpathy
41d5f56278
might as well push the few kernels that i feel ok about so far
2024-04-09 14:34:13 +00:00
