26 commits

Author	SHA1	Message	Date
Erik Schultheis	cb8cc25e3a	utilities for Packed and more vectorization	2024-05-27 13:15:36 +03:00
Andrej Karpathy	2d43e5bc97	remove legacy comment	2024-05-14 19:14:07 +00:00
Erik Schultheis	e553e2f084	update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results	2024-05-14 20:44:50 +03:00
ngc92	5920143d47	remove warning noise	2024-05-10 00:02:37 +02:00
ngc92	691c1df969	fused layernorm+residual	2024-05-09 23:48:11 +02:00
ngc92	99e765de8d	moved bf16 boilerplate to common.h	2024-05-09 11:19:24 +02:00
ademeure	1ea7f9bf25	tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changes	2024-05-07 00:54:50 +01:00
Erik Schultheis	06980b6b2f	mixed precision utilities for dev/cuda	2024-05-01 22:35:55 +03:00
Andrej Karpathy	769c911ab6	small tweaks	2024-04-30 23:48:06 +00:00
Erik Schultheis	c3515cfe47	fixed potential error and generalized gelu forward	2024-04-30 23:45:34 +03:00
Andrej Karpathy	7ba1ed8a1a	delete packed128 .fp32, use (float) ok	2024-04-30 19:39:20 +00:00
Erik Schultheis	d7813d281d	clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results	2024-04-30 12:45:37 +03:00
Andrej Karpathy	af2bc47d22	remove forceinline. blame @arund42 if this ever does not get inlined by the compiler	2024-04-29 23:10:51 +00:00
Andrej Karpathy	5e12c66f5a	move packed128 to common	2024-04-29 22:55:28 +00:00
ademeure	49ffd68317	Fix sizeof(T) -> sizeof(D) (was overallocating memory)	2024-04-29 15:47:59 +01:00
ademeure	82fef03acd	Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)	2024-04-29 01:51:10 +01:00
Erik Schultheis	732a8b4806	added NaN-based makes for reference checks	2024-04-22 14:53:24 +03:00
Erik Schultheis	16123c3af8	towards an even better backward attention kernel	2024-04-19 19:12:30 +03:00
Andrej Karpathy	c393d828dc	size_t instead of int. everywhere. this will come to bite us at one point...	2024-04-19 02:11:30 +00:00
Andrej Karpathy	8a673edcc3	push my attention_backward file	2024-04-17 04:42:42 +00:00
Andrej Karpathy	6be8080501	add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far	2024-04-16 02:10:45 +00:00
Andrej Karpathy	c5ee371564	make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common	2024-04-13 17:56:18 +00:00
Andrej Karpathy	84145411d3	fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans	2024-04-13 17:47:19 +00:00
Erik Schultheis	f459bfa833	fixed re-initialization of device memory after test	2024-04-13 18:23:12 +03:00
Erik Schultheis	59bada94d9	improved testing	2024-04-13 17:06:52 +03:00
Erik Schultheis	7e8e038de8	extracted common testing functions into their own header	2024-04-13 13:00:05 +03:00