aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-25 20:05:08 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

23 commits

Author SHA1 Message Date
Andrej
109f516367
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs 
Free cooperative groups implementation of online softmax forward
 2024-05-16 21:23:56 +01:00
lancer
5c90845f7a update the -lcublas -lcublasLt flag in the comment 2024-05-09 22:04:02 -07:00
ngc92
691c1df969 fused layernorm+residual 2024-05-09 23:48:11 +02:00
KarhouTam
6d7a99cb4e Adjust comments 2024-05-08 17:45:14 +08:00
KarhouTam
6a52d8619e Optimize codes and comments 2024-05-08 16:12:39 +08:00
KarhouTam
b3e8a9fe67 Implementation of online softmax forward kernel without cgs. 2024-05-08 09:22:14 +08:00
lancer
e991fe1efa fix the 0.0f initialization for double 2024-04-18 19:33:30 -07:00
Erik Schultheis
c73fa8842a fixed numerics for online kernel, and an existing race condition in kernel 2 2024-04-15 03:41:09 +03:00
Erik Schultheis
8fb966bc31 kernel 1 should also accumulate in double to ensure stability 2024-04-15 02:52:38 +03:00
Erik Schultheis
60cf281e46 use V dimension in softmax tests 
adjusted tests to be more challenging
 2024-04-15 02:14:48 +03:00
Erik Schultheis
99b42c1a94 nicer printing 2024-04-15 02:03:48 +03:00
Andrej Karpathy
534b93a732 erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7 2024-04-13 22:02:31 +00:00
Erik Schultheis
59bada94d9 improved testing 2024-04-13 17:06:52 +03:00
Erik Schultheis
7e7460cfbd const'ness fixes 2024-04-13 13:38:11 +03:00
Erik Schultheis
7e8e038de8 extracted common testing functions into their own header 2024-04-13 13:00:05 +03:00
Andrej Karpathy
40b78be722 small changes to comments, and very minor code modifications and movements 2024-04-13 02:11:36 +00:00
lancer
69d0e5bac7 Include a parallel kernal of softmax_online 2024-04-11 15:30:36 -07:00
lancer
ca1e5ee04d Include a parallel kernal of softmax_online 2024-04-11 15:24:41 -07:00
lancer
d6bcc129b9 Improve the efficiency 2024-04-11 12:16:20 -07:00
lancer
00010c7188 Improve the efficiency 2024-04-11 12:13:34 -07:00
lancer
f524a90de3 Include the online softmax CPU code (from the paper "Online normalizer calculation for softmax") and native port to GPU kernel. 
Both the CPU and GPU kernels' correctness are verified by executing the script.

Performance: the online kernel (native CPU port) is faster than the kernel1 (native CPU port) but worse than kernel 2-4.
 2024-04-11 09:47:50 -07:00
Andrej Karpathy
8386e5393c intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice 2024-04-09 20:08:11 +00:00
Andrej Karpathy
c29d70a9b3 add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions 2024-04-09 19:22:47 +00:00