28 commits

Author	SHA1	Message	Date
lancer	5c90845f7a	update the -lcublas -lcublasLt flag in the comment	2024-05-09 22:04:02 -07:00
ngc92	99e765de8d	moved bf16 boilerplate to common.h	2024-05-09 11:19:24 +02:00
Erik Schultheis	91bc72d0c0	fixup	2024-05-01 22:40:26 +03:00
Erik Schultheis	06980b6b2f	mixed precision utilities for dev/cuda	2024-05-01 22:35:55 +03:00
Andrej Karpathy	769c911ab6	small tweaks	2024-04-30 23:48:06 +00:00
Erik Schultheis	603d862830	fix up bandwidth calculation	2024-04-30 23:52:08 +03:00
Erik Schultheis	c3515cfe47	fixed potential error and generalized gelu forward	2024-04-30 23:45:34 +03:00
Andrej Karpathy	9141e04693	fixes to the PR, careful with float/floatX etc	2024-04-30 19:35:05 +00:00
Christopher Dryden	568615fef1	Added packing for gelu forwards kernel	2024-04-30 03:21:42 +00:00
Andrej Karpathy	5e12c66f5a	move packed128 to common	2024-04-29 22:55:28 +00:00
Andrej Karpathy	f43592954f	reset back to precious hyperparameters	2024-04-29 22:44:25 +00:00
Andrej Karpathy	ee125cffc4	make it imo clearer that the grid size is one quarter as large as it would otherwise be. because each thread now does 4X work	2024-04-29 22:40:26 +00:00
Andrej Karpathy	083b210dd4	small changes	2024-04-29 22:38:03 +00:00
Andrej Karpathy	d9506a7a0a	minor changes to packed128	2024-04-29 22:31:19 +00:00
Erik Schultheis	4872c57164	comments	2024-04-29 22:46:33 +03:00
Erik Schultheis	9f07a176e0	simplify to only support cs	2024-04-29 22:43:52 +03:00
Erik Schultheis	896b644cb9	slight simplification	2024-04-29 22:19:12 +03:00
Erik Schultheis	45252d434c	yet another gelu	2024-04-29 21:53:08 +03:00
Erik Schultheis	99b42c1a94	nicer printing	2024-04-15 02:03:48 +03:00
Erik Schultheis	fe1d8cfb47	fix timing output	2024-04-15 02:03:42 +03:00
Andrej Karpathy	82623518a4	fix comment block of gelu forward kernel	2024-04-13 17:48:53 +00:00
Erik Schultheis	59bada94d9	improved testing	2024-04-13 17:06:52 +03:00
Erik Schultheis	7e7460cfbd	const'ness fixes	2024-04-13 13:38:11 +03:00
Erik Schultheis	7e8e038de8	extracted common testing functions into their own header	2024-04-13 13:00:05 +03:00
Andrej Karpathy	f26cf00a61	make gelu constant be a #define, speeds up the kernel by 1% or so	2024-04-11 15:49:01 +00:00
lancer	3a4d783633	amend	2024-04-09 09:59:03 -07:00
lancer	73c92912b0	fix error: identifier M_PI is undefined	2024-04-09 09:51:28 -07:00
Andrej Karpathy	41d5f56278	might as well push the few kernels that i feel ok about so far	2024-04-09 14:34:13 +00:00