292 commits

Author	SHA1	Message	Date
Erik Schultheis	d6793645f5	replace cublaslt with custom matmul	2024-06-12 09:42:19 +03:00
Erik Schultheis	7f3f3acf16	shared memory matmul	2024-06-11 21:03:43 +03:00
Erik Schultheis	071c977c79	vectorized memory access	2024-06-11 00:46:51 +03:00
Erik Schultheis	bb89efdcb1	register reuse	2024-06-11 00:41:08 +03:00
Andrej	637c1b69c4	Merge pull request #552 from karpathy/feature/streams ... Feature/streams	2024-06-08 09:25:56 -07:00
Andrej Karpathy	ee6b3c9325	take out Async copies and memsets	2024-06-08 16:17:02 +00:00
Aleksa Gordic	d3790c64a1	Change default path	2024-06-07 17:38:01 +02:00
Aleksa Gordic	4090ca8fa1	Add learning rate and grad norm logging	2024-06-07 17:34:42 +02:00
Aleksa Gordic	7b99ac96a5	Fix mem leaks, reduce memory, refactor	2024-06-06 19:58:50 +02:00
Andrej	d931fc0507	-std=c++17 default, merge PR #559 from ngc92/ubuntu20.04 ... set compile flag and add ci check	2024-06-06 09:23:07 -07:00
Erik Schultheis	875196b672	add c++17 flag also to dev/cuda	2024-06-06 19:02:45 +03:00
Aleksa Gordic	c116fbfa29	Fix q,k bug, move all kernels to q@k^t form	2024-06-05 22:22:55 +02:00
Aleksa Gordic	875d362381	Refactor trimat	2024-06-05 17:33:26 +02:00
Andrej	80e0c3b8d9	Merge pull request #512 from gordicaleksa/refactor_encoder_bwd_kernel ... Remove redundant CPU computation in encoder bwd	2024-06-02 16:43:10 -07:00
Aleksa Gordic	290c00a362	Remove redundant CPU computation	2024-06-01 18:27:24 +02:00
Andrej	a26041af12	Merge pull request #495 from ChrisDryden/shared_memory ... Removed unnecesary shared memory due to blockreduce using static defined shared memory	2024-06-01 08:12:21 -07:00
Aleksa Gordic	8eafd40615	Update compile cmd in the dev/cuda README	2024-06-01 16:46:32 +02:00
Andrej Karpathy	f8f9e49ba3	Merge branch 'master' into feature/libdir	2024-05-30 19:07:51 +00:00
Christopher	5450632237	Removed unnecesary shared memory due to blockreduce using static defined shared memory	2024-05-30 03:01:52 +00:00
Andrej	55f3665b64	Merge pull request #493 from vyom1611/master-2 ... Modal benchmarking script updated to replace deprecated calls	2024-05-29 07:50:10 -07:00
vyom1611	062f096da1	Modal benchmarking script updated to replace deprecated calls	2024-05-29 20:01:41 +05:30
Andrej	2db7716d68	Merge pull request #476 from lancerts/layernorm-bk-fix ... Fix "Mismatch of dweight at layernorm_backward.cu #428"	2024-05-29 07:30:52 -07:00
Andrej Karpathy	0c83a23188	tweak the logfile parser and visualizer	2024-05-28 01:38:15 +00:00
lancerts	47c670b885	amend	2024-05-27 15:13:11 -07:00
lancerts	c4c985cb87	spell explicitly uint to unsigned int	2024-05-27 14:44:47 -07:00
lancerts	f27ca4df41	fix the issue Mismatch of dweight at layernorm_backward.cu	2024-05-27 14:39:57 -07:00
Ross Wheeler	bb80f93618	Minor cleanup ... Added static inline to the functions	2024-05-27 13:16:34 -07:00
Ross Wheeler	758acdffbf	Adding dirent.h support to unistd.h	2024-05-27 13:08:07 -07:00
Erik Schultheis	7dc3b7d7dc	bugfix	2024-05-27 16:36:44 +03:00
Erik Schultheis	5f73ecfcf1	fix C % 256 != 0	2024-05-27 13:15:36 +03:00
Erik Schultheis	d35daf19c9	fail fast and hard; don't go into the deadlock	2024-05-27 13:15:36 +03:00
Erik Schultheis	3b6808269a	some comments and optimized shared memory amount	2024-05-27 13:15:36 +03:00
Erik Schultheis	9eba86f0a5	fully vectorized smem access	2024-05-27 13:15:36 +03:00
Erik Schultheis	cb8cc25e3a	utilities for Packed and more vectorization	2024-05-27 13:15:36 +03:00
Erik Schultheis	53ee3297dd	use vectorized access to shared memory	2024-05-27 13:15:36 +03:00
Andrej Karpathy	ee6fa5ec9f	fill in the other model sizes	2024-05-26 18:04:43 +00:00
Andrej Karpathy	5ec4b774f2	simple jupyter notebook example on how to parse and vis the logfiles	2024-05-26 17:40:49 +00:00
Andrej Karpathy	d0c014372c	add ability to export fineweb100B as well	2024-05-25 17:21:51 +00:00
Andrej Karpathy	2a0f78d617	attempt to fix windows episode 2	2024-05-25 00:44:16 +00:00
Andrej Karpathy	5b96d91893	attempt to fix windows?	2024-05-25 00:39:40 +00:00
Andrej	4ff0412751	Merge pull request #450 from rosslwheeler/dataloader_win_fixes ... Add glob() for windows	2024-05-24 17:30:35 -07:00
Andrej	6a7fd56d7c	Merge pull request #439 from lancerts/matmul-fix ... Fix the unsupported block_size in matmul_backward_bias kernel 1	2024-05-24 17:26:37 -07:00
Ross Wheeler	79738d2ca4	fixed mkdir change	2024-05-24 17:05:59 -07:00
Ross Wheeler	e5083be900	Moved windows glob() over to dev/unistd.h ... Added header guard and changed long->int64_t in dataloader.h	2024-05-24 16:56:52 -07:00
Andrej Karpathy	dbacaf84cf	Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm	2024-05-24 21:46:25 +00:00
Andrej	bf03e7fd4c	Merge pull request #456 from ngc92/large-batch-fix ... fix for large batch sizes	2024-05-24 12:00:41 -07:00
Erik Schultheis	1b98637960	int -> int64_t	2024-05-24 20:11:34 +03:00
Andrej Karpathy	08fe3eef43	change default shard size for fineweb10B	2024-05-24 15:56:05 +00:00
Erik Schultheis	b66eb66ac7	update fine-web preprocessing: faster, and much less RAM consumption	2024-05-24 17:45:26 +03:00
Andrej Karpathy	da59861d38	make hellaswag optional eval yay	2024-05-22 22:42:37 +00:00