13 commits

Author	SHA1	Message	Date
Erik Schultheis	06980b6b2f	mixed precision utilities for dev/cuda	2024-05-01 22:35:55 +03:00
patricxu	d48b0d2b2b	fix indexing in matmul_forward_kernel1	2024-04-20 15:38:12 +08:00
Erik Schultheis	99b42c1a94	nicer printing	2024-04-15 02:03:48 +03:00
Andrej Karpathy	c5ee371564	make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common	2024-04-13 17:56:18 +00:00
Erik Schultheis	59bada94d9	improved testing	2024-04-13 17:06:52 +03:00
Erik Schultheis	7e7460cfbd	const'ness fixes	2024-04-13 13:38:11 +03:00
Erik Schultheis	7e8e038de8	extracted common testing functions into their own header	2024-04-13 13:00:05 +03:00
Andrej	8822e7803e	use cublaslt, which fuses bias, and also use tf32 when your GPU supports it (#98) ... Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.	2024-04-12 15:32:07 -07:00
lancer	68d2ce0601	fix error in small typos in matmul_forward.cu	2024-04-10 13:35:20 -07:00
Andrej	9f5069d3b2	Merge pull request #38 from lancerts/fix_matmul ... fix the consistency of the transpose notation in matmul_foward.cu	2024-04-10 12:15:53 -07:00
Andrej Karpathy	80f52e5290	the full forward pass of GPT-2 in one file of pure CUDA	2024-04-10 18:15:55 +00:00
lancer	fbd679adec	fix the consistency of the transpose notation	2024-04-09 15:19:55 -07:00
Andrej Karpathy	41d5f56278	might as well push the few kernels that i feel ok about so far	2024-04-09 14:34:13 +00:00