15 commits

Author	SHA1	Message	Date
lancer	5c90845f7a	update the -lcublas -lcublasLt flag in the comment	2024-05-09 22:04:02 -07:00
Andrej Karpathy	91aab62aad	fix kernel 5 of layernorm, bug from previous copy paste with loop bounds	2024-04-20 00:51:06 +00:00
Andrej Karpathy	49d41ae296	add one more kernel, allocating a block per row. bad idea if C is too low, as we have it right now	2024-04-19 23:24:51 +00:00
Andrej Karpathy	cb791c4ef5	new kernel that does a single pass over x on load, using a more clever variance formula. only very slightly faster on my A100 sadly	2024-04-19 22:55:30 +00:00
Erik Schultheis	99b42c1a94	nicer printing	2024-04-15 02:03:48 +03:00
Erik Schultheis	59bada94d9	improved testing	2024-04-13 17:06:52 +03:00
Erik Schultheis	7e7460cfbd	const'ness fixes	2024-04-13 13:38:11 +03:00
Erik Schultheis	7e8e038de8	extracted common testing functions into their own header	2024-04-13 13:00:05 +03:00
Andrej Karpathy	00b9fda4a1	ceildiv in layernorm	2024-04-11 18:21:34 +00:00
Andrej Karpathy	089cd5f1dd	make sure we also test correctness of mean and rstd and do it all block sizes	2024-04-11 18:12:26 +00:00
Andrej Karpathy	f28613e0b9	small edits to comments and variables	2024-04-11 18:09:54 +00:00
Erik Schultheis	4e91362fa2	optionally generate mean and rstd	2024-04-11 20:48:14 +03:00
Erik Schultheis	afddb1af5b	fixed block size and improved load op hint	2024-04-11 03:47:38 +03:00
Erik Schultheis	466723bb06	fully fused layer-norm kernel	2024-04-11 02:01:41 +03:00
Andrej Karpathy	41d5f56278	might as well push the few kernels that i feel ok about so far	2024-04-09 14:34:13 +00:00