1022 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Andrej Karpathy	49b9776e50	also move the launchers	2024-06-03 21:37:18 +00:00
Andrej Karpathy	93546aed01	move encoder, kernel 1/N	2024-06-03 21:21:28 +00:00
Andrej	3392671826	Merge pull request #535 from ngc92/speed-up-compile ... moved device utilities into a separate file, and follow-up cleanup	2024-06-03 14:11:10 -07:00
Erik Schultheis	67502c7f8e	moved device utilities into a separate file, and follow-up cleanup	2024-06-03 23:58:47 +03:00
Andrej	8cc39a794b	Merge pull request #528 from gordicaleksa/refactor_evalloader ... Refactor eval loader - remove 2 unnecessary malloc calls per iteration	2024-06-03 13:54:52 -07:00
Andrej	2a6a5b2c6c	Merge pull request #529 from gordicaleksa/improve_start_training_print_info ... Improve start training print info	2024-06-03 13:53:32 -07:00
Andrej	0cd9f1ac21	Merge pull request #530 from gordicaleksa/minor_refactor ... Minor refactor: remove hardcoded val, delete unused vars...	2024-06-03 13:52:42 -07:00
Andrej	d17893b754	Merge pull request #532 from ngc92/ln-buffers ... more efficient use of memory buffers for LN recomputation	2024-06-03 13:51:22 -07:00
Andrej	ac931455a4	Merge pull request #533 from karpathy/feature/refactor_part2 ... refactor part 2	2024-06-03 13:44:59 -07:00
Andrej Karpathy	001ad78fe1	resolve deviceProp and cudaCheck dupes	2024-06-03 19:54:32 +00:00
Andrej Karpathy	020b15ad77	small more fixes	2024-06-03 19:32:32 +00:00
Andrej Karpathy	5d9403479c	some fixes, still not done	2024-06-03 18:50:41 +00:00
Andrej Karpathy	804d0693c8	refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well	2024-06-03 18:10:33 +00:00
Andrej	e40268457f	Merge pull request #526 from karpathy/try/move_cudnn_to_llmc ... attempt to move cudnn attention to the llmc directory	2024-06-03 09:32:31 -07:00
Erik Schultheis	0d16b51626	more efficient use of memory buffers for LN recomputation	2024-06-03 18:32:51 +03:00
Andrej Karpathy	6434009298	attempt to remove PATH_SEP from Makefile	2024-06-03 15:30:45 +00:00
Aleksa Gordic	42ca620306	Minor refactor: remove hardcoded val, delete unused vars...	2024-06-03 16:24:22 +02:00
Aleksa Gordic	7413a1a315	Remove unnecessary load filename line	2024-06-03 12:33:33 +02:00
Aleksa Gordic	728762ceed	Improve start of training print info	2024-06-03 12:31:33 +02:00
Aleksa Gordic	d9e867678e	Remove file size - not used	2024-06-03 12:14:58 +02:00
Aleksa Gordic	e86bd62374	Refactor eval loader - remove 2 unnecessary malloc calls per iteration	2024-06-03 11:43:33 +02:00
Andrej Karpathy	41dc65a0ef	attempt to move cudnn attention to the llmc directory	2024-06-03 03:23:04 +00:00
Andrej Karpathy	08fc45b492	delete copy paste code for gpt2 model init	2024-06-03 02:08:29 +00:00
Andrej Karpathy	c827dd2f8e	Merge branch 'ngc92-mfu'	2024-06-03 01:41:31 +00:00
Andrej Karpathy	1ac30591f2	add new file llmc/mfu.h for MFU calculations for different GPUs	2024-06-03 01:41:18 +00:00
Andrej	80e0c3b8d9	Merge pull request #512 from gordicaleksa/refactor_encoder_bwd_kernel ... Remove redundant CPU computation in encoder bwd	2024-06-02 16:43:10 -07:00
Andrej	cb5eff0578	Merge pull request #514 from ngc92/compile-fix ... fix compilation with older nvcc	2024-06-02 16:42:31 -07:00
Andrej	ff98d8bda7	Merge pull request #515 from gordicaleksa/fix_adamw_kernel3_num_threads ... Use local params for num blocks in adamw_kernel3	2024-06-02 16:41:59 -07:00
Andrej	1ce30fdc4a	Merge pull request #516 from gordicaleksa/update_random_doc ... Add clarification on Box-Muller	2024-06-02 16:37:37 -07:00
Andrej	252584954e	Merge pull request #519 from gordicaleksa/mem_leak ... Fix mem leak	2024-06-02 16:34:54 -07:00
Andrej Karpathy	1e893f4e11	Merge branch 'gordicaleksa-fix_zero_stage1'	2024-06-02 23:28:22 +00:00
Andrej Karpathy	4944ac5743	adjust comments	2024-06-02 23:28:13 +00:00
Aleksa Gordic	a8a5a444bc	Add ; - ah c my old friend :)	2024-06-02 23:04:47 +02:00
Aleksa Gordic	9a455edc25	Improve comments	2024-06-02 23:03:41 +02:00
Aleksa Gordic	b4e8bd7f66	Fix zero stage 1 norm computation - WIP	2024-06-02 22:45:26 +02:00
Aleksa Gordic	df0a826ef3	Close file	2024-06-02 21:37:52 +02:00
Aleksa Gordic	30d81cce15	Fix mem leak	2024-06-02 21:28:42 +02:00
Aleksa Gordic	79cbc03ccf	Use local params for num blocks	2024-06-02 15:06:06 +02:00
Aleksa Gordic	ae9c957273	Add clarification on Box-Muller	2024-06-02 15:04:45 +02:00
Erik Schultheis	a902323f36	fix compilation with older nvcc	2024-06-02 12:40:28 +03:00
Aleksa Gordic	290c00a362	Remove redundant CPU computation	2024-06-01 18:27:24 +02:00
Andrej	a26041af12	Merge pull request #495 from ChrisDryden/shared_memory ... Removed unnecesary shared memory due to blockreduce using static defined shared memory	2024-06-01 08:12:21 -07:00
Andrej	9cf8c2f135	Merge pull request #509 from ngc92/inplace ... Inplace gelu backward	2024-06-01 08:10:56 -07:00
Andrej	bd0f036ee5	Merge pull request #511 from gordicaleksa/gordicaleksa_fix_docs ... Update compile cmd in the dev/cuda README	2024-06-01 08:09:31 -07:00
Aleksa Gordic	8eafd40615	Update compile cmd in the dev/cuda README	2024-06-01 16:46:32 +02:00
Erik Schultheis	761fb2cf2b	rename gelu_backward to gelu_backward_inplace, and made it obvious in the code that this is an inplace operation	2024-06-01 14:31:26 +03:00
Erik Schultheis	cc4bed0806	added missing checks	2024-06-01 14:30:40 +03:00
Andrej Karpathy	9c8edaffa7	fix recompute bug	2024-05-31 22:38:03 +00:00
Andrej Karpathy	0bc24c36e6	fix bug in resumption of dataloader, have to reload shard	2024-05-31 22:19:37 +00:00
Andrej Karpathy	1de3bef0ac	fix argparse comment too	2024-05-31 22:06:48 +00:00