370 commits

Author	SHA1	Message	Date
Andrej Karpathy	49b9776e50	also move the launchers	2024-06-03 21:37:18 +00:00
Andrej Karpathy	93546aed01	move encoder, kernel 1/N	2024-06-03 21:21:28 +00:00
Andrej	3392671826	Merge pull request #535 from ngc92/speed-up-compile ... moved device utilities into a separate file, and follow-up cleanup	2024-06-03 14:11:10 -07:00
Erik Schultheis	67502c7f8e	moved device utilities into a separate file, and follow-up cleanup	2024-06-03 23:58:47 +03:00
Andrej	2a6a5b2c6c	Merge pull request #529 from gordicaleksa/improve_start_training_print_info ... Improve start training print info	2024-06-03 13:53:32 -07:00
Andrej	0cd9f1ac21	Merge pull request #530 from gordicaleksa/minor_refactor ... Minor refactor: remove hardcoded val, delete unused vars...	2024-06-03 13:52:42 -07:00
Andrej	d17893b754	Merge pull request #532 from ngc92/ln-buffers ... more efficient use of memory buffers for LN recomputation	2024-06-03 13:51:22 -07:00
Andrej Karpathy	001ad78fe1	resolve deviceProp and cudaCheck dupes	2024-06-03 19:54:32 +00:00
Andrej Karpathy	804d0693c8	refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well	2024-06-03 18:10:33 +00:00
Erik Schultheis	0d16b51626	more efficient use of memory buffers for LN recomputation	2024-06-03 18:32:51 +03:00
Aleksa Gordic	42ca620306	Minor refactor: remove hardcoded val, delete unused vars...	2024-06-03 16:24:22 +02:00
Aleksa Gordic	7413a1a315	Remove unnecessary load filename line	2024-06-03 12:33:33 +02:00
Aleksa Gordic	728762ceed	Improve start of training print info	2024-06-03 12:31:33 +02:00
Andrej Karpathy	08fc45b492	delete copy paste code for gpt2 model init	2024-06-03 02:08:29 +00:00
Andrej Karpathy	1ac30591f2	add new file llmc/mfu.h for MFU calculations for different GPUs	2024-06-03 01:41:18 +00:00
Andrej	cb5eff0578	Merge pull request #514 from ngc92/compile-fix ... fix compilation with older nvcc	2024-06-02 16:42:31 -07:00
Andrej	ff98d8bda7	Merge pull request #515 from gordicaleksa/fix_adamw_kernel3_num_threads ... Use local params for num blocks in adamw_kernel3	2024-06-02 16:41:59 -07:00
Andrej	252584954e	Merge pull request #519 from gordicaleksa/mem_leak ... Fix mem leak	2024-06-02 16:34:54 -07:00
Andrej Karpathy	4944ac5743	adjust comments	2024-06-02 23:28:13 +00:00
Aleksa Gordic	a8a5a444bc	Add ; - ah c my old friend :)	2024-06-02 23:04:47 +02:00
Aleksa Gordic	9a455edc25	Improve comments	2024-06-02 23:03:41 +02:00
Aleksa Gordic	b4e8bd7f66	Fix zero stage 1 norm computation - WIP	2024-06-02 22:45:26 +02:00
Aleksa Gordic	df0a826ef3	Close file	2024-06-02 21:37:52 +02:00
Aleksa Gordic	30d81cce15	Fix mem leak	2024-06-02 21:28:42 +02:00
Aleksa Gordic	79cbc03ccf	Use local params for num blocks	2024-06-02 15:06:06 +02:00
Erik Schultheis	a902323f36	fix compilation with older nvcc	2024-06-02 12:40:28 +03:00
Andrej	a26041af12	Merge pull request #495 from ChrisDryden/shared_memory ... Removed unnecesary shared memory due to blockreduce using static defined shared memory	2024-06-01 08:12:21 -07:00
Andrej	9cf8c2f135	Merge pull request #509 from ngc92/inplace ... Inplace gelu backward	2024-06-01 08:10:56 -07:00
Erik Schultheis	761fb2cf2b	rename gelu_backward to gelu_backward_inplace, and made it obvious in the code that this is an inplace operation	2024-06-01 14:31:26 +03:00
Erik Schultheis	cc4bed0806	added missing checks	2024-06-01 14:30:40 +03:00
Andrej Karpathy	9c8edaffa7	fix recompute bug	2024-05-31 22:38:03 +00:00
Andrej Karpathy	0bc24c36e6	fix bug in resumption of dataloader, have to reload shard	2024-05-31 22:19:37 +00:00
Andrej Karpathy	1de3bef0ac	fix argparse comment too	2024-05-31 22:06:48 +00:00
Andrej Karpathy	f4874a9f63	make things a bit more consistent, keep recompute 1 as default, and add a comment	2024-05-31 22:04:34 +00:00
Andrej Karpathy	629a24020d	Merge branch 'layernorm_backwards_mem_saving' of https://github.com/ChrisDryden/llm.c into ChrisDryden-layernorm_backwards_mem_saving	2024-05-31 21:47:28 +00:00
Andrej Karpathy	35aa110244	more refactor	2024-05-30 19:39:29 +00:00
Andrej Karpathy	f8f9e49ba3	Merge branch 'master' into feature/libdir	2024-05-30 19:07:51 +00:00
Christopher	59432fea19	modified the recalculation parameter to be based off of 2 instead of 1	2024-05-30 17:23:47 +00:00
Christopher	e5719e18b3	Rearranged layernorm forward for 1 and 2 and used layer variables instead. Validated with test file	2024-05-30 16:28:03 +00:00
Christopher	95b1fad088	First step in creating option to recalculate layernorm activations in backwards pass	2024-05-30 16:28:03 +00:00
Chris Dryden	3bca31fde1	Nit: duplicated line in header of project	2024-05-29 20:03:13 -07:00
Christopher	5450632237	Removed unnecesary shared memory due to blockreduce using static defined shared memory	2024-05-30 03:01:52 +00:00
Andrej Karpathy	4cc8b1570b	hardcode grad_clip to 1.0, delete confusion around legacy code that used -c flag for it, but became inactive in the if else block	2024-05-29 19:01:12 +00:00
QIAOQIAN	27cf5223e0	softmax_autoregressive_backward_kernel does not use share memory in the kernel ... we do not need to launch the kernel with 256 bytes share memory, so remove it	2024-05-29 09:23:46 +08:00
Erik Schultheis	daa95e04ae	ampere-consumer	2024-05-29 02:27:11 +03:00
Erik Schultheis	de7a9b83cd	more GPUs	2024-05-29 02:20:31 +03:00
Erik Schultheis	184cc9d10a	added H100 SMX version	2024-05-29 01:31:16 +03:00
Erik Schultheis	e2b411362e	include string view	2024-05-29 00:54:23 +03:00
Erik Schultheis	9374d751de	MFU for other GPUs	2024-05-29 00:48:37 +03:00
Andrej Karpathy	037ec67a5d	small tweak to launch	2024-05-28 15:07:07 +00:00