242 commits

Author	SHA1	Message	Date
Andrej Karpathy	dcff5613c6	add kernel3, which is like kernel 2 but no float4	2024-04-19 01:19:20 +00:00
Andrej Karpathy	8a2cab3431	make default dloss be 1/BT	2024-04-19 00:27:28 +00:00
Andrej Karpathy	ea3e2abdcf	small fixes and more comments very cool kernels	2024-04-18 23:48:39 +00:00
Erik Schultheis	09d35a5e4d	splitting the loop to get rid of the indicator	2024-04-19 01:01:47 +03:00
Erik Schultheis	6d98af1d32	another 15% by getting rid of ifs	2024-04-19 01:01:47 +03:00
Erik Schultheis	314661f516	significantly faster kernel by reusing data in registers	2024-04-19 01:01:47 +03:00
Andrej Karpathy	de1e87158c	add comments to classifier_fused	2024-04-18 16:59:45 +00:00
Andrej	2d2f1df8f1	Merge pull request #150 from ademeure/optimised_fused_classifier ... Optimised version of fused classifier + bugfixes(?)	2024-04-18 08:39:05 -07:00
FeSens	0c09a37568	fix(attention_forward.cu): Fix the number of elements to check when verifying attention implementations. ... This was causing me problems when I was trying to debug a new implementation on fewer tokens.	2024-04-18 01:23:05 -03:00
lancer	a1abcbc32d	fix the bug that yields cpu, gpu results mismatch	2024-04-17 21:04:45 -07:00
ademeure	557fecd6e9	bugfix for bounds checking	2024-04-18 04:45:07 +01:00
ademeure	2fd8ada4a6	Merge remote-tracking branch 'karpathy/master' into optimised_fused_classifier	2024-04-18 03:51:01 +01:00
ademeure	ea77a26473	mini fix	2024-04-18 03:42:34 +01:00
ademeure	3ba2cdc723	fixes + bounds checking	2024-04-18 03:41:27 +01:00
lancer	ac055f730c	fix the bug that yields cpu, gpu results mismatch	2024-04-17 17:47:11 -07:00
Erik Schultheis	58a7b57f96	enable benchmarking	2024-04-18 01:39:47 +03:00
Erik Schultheis	e278112c19	coalesced reading in attention backward pass	2024-04-18 01:39:43 +03:00
ademeure	4ee98f5135	Merge branch 'master' into optimised_fused_classifier	2024-04-17 22:58:51 +01:00
lancer	c327923df9	Constness fix layernorm_backward.cu	2024-04-17 14:56:03 -07:00
lancer	81aa596251	Constness fix layernorm_backward.cu	2024-04-17 14:54:49 -07:00
Erik Schultheis	c1bfe4a639	warp-wide reduction for layernorm backward	2024-04-17 21:52:40 +03:00
Andrej Karpathy	fa4ebfa663	kernel2 for softmax backward	2024-04-17 08:14:22 +00:00
Andrej Karpathy	d74d7ae652	small adamw fixes, needs more work	2024-04-17 04:58:16 +00:00
Andrej Karpathy	66a0b5242e	Merge branch 'master' of github.com:karpathy/llm.c	2024-04-17 04:51:26 +00:00
Andrej Karpathy	8a673edcc3	push my attention_backward file	2024-04-17 04:42:42 +00:00
nopperl	97e61e2258	add slightly more optimized adamw kernel	2024-04-17 03:31:07 +02:00
nopperl	5684ff7e77	add adamw cuda kernel	2024-04-17 03:30:59 +02:00
Erik Schultheis	43bf9eb66a	ensure valid test cases and incorporate fixes by @ademeure	2024-04-16 20:53:50 +03:00
ademeure	b225501b7b	optimised version of the cool new fused classifier + hopefully bugfixes?	2024-04-16 14:54:23 +01:00
lancer	6ff4ad646d	fix the backward dbias precision issue	2024-04-15 19:58:16 -07:00
Andrej Karpathy	6be8080501	add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far	2024-04-16 02:10:45 +00:00
Andrej Karpathy	3de507739c	minor fixes to matmul backward	2024-04-16 01:23:41 +00:00
Andrej Karpathy	9560b1a92b	add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at first	2024-04-16 00:40:11 +00:00
Andrej	3c6dafeb53	Merge pull request #117 from ngc92/fused-classifier ... WIP: Fully fused classification layer	2024-04-15 12:13:48 -07:00
Erik Schultheis	c73fa8842a	fixed numerics for online kernel, and an existing race condition in kernel 2	2024-04-15 03:41:09 +03:00
Erik Schultheis	8fb966bc31	kernel 1 should also accumulate in double to ensure stability	2024-04-15 02:52:38 +03:00
Erik Schultheis	60cf281e46	use V dimension in softmax tests ... adjusted tests to be more challenging	2024-04-15 02:14:48 +03:00
Erik Schultheis	99b42c1a94	nicer printing	2024-04-15 02:03:48 +03:00
Erik Schultheis	fe1d8cfb47	fix timing output	2024-04-15 02:03:42 +03:00
Erik Schultheis	780d8bd012	initial version	2024-04-14 03:48:56 +03:00
Andrej Karpathy	534b93a732	erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7	2024-04-13 22:02:31 +00:00
Andrej Karpathy	c5ee371564	make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common	2024-04-13 17:56:18 +00:00
Andrej Karpathy	82623518a4	fix comment block of gelu forward kernel	2024-04-13 17:48:53 +00:00
Andrej Karpathy	02b5b33c5f	remove unused variable C	2024-04-13 17:48:11 +00:00
Andrej Karpathy	84145411d3	fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans	2024-04-13 17:47:19 +00:00
Andrej Karpathy	69f0385237	add comments and handling of kernels and their correctness	2024-04-13 17:44:31 +00:00
Erik Schultheis	f459bfa833	fixed re-initialization of device memory after test	2024-04-13 18:23:12 +03:00
Erik Schultheis	d035725d22	mark problematic tests	2024-04-13 18:09:28 +03:00
Erik Schultheis	59bada94d9	improved testing	2024-04-13 17:06:52 +03:00
Erik Schultheis	7e7460cfbd	const'ness fixes	2024-04-13 13:38:11 +03:00