1007 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Andrej Karpathy	001ad78fe1	resolve deviceProp and cudaCheck dupes	2024-06-03 19:54:32 +00:00
Andrej Karpathy	020b15ad77	small more fixes	2024-06-03 19:32:32 +00:00
Andrej Karpathy	5d9403479c	some fixes, still not done	2024-06-03 18:50:41 +00:00
Andrej Karpathy	804d0693c8	refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well	2024-06-03 18:10:33 +00:00
Andrej	e40268457f	Merge pull request #526 from karpathy/try/move_cudnn_to_llmc ... attempt to move cudnn attention to the llmc directory	2024-06-03 09:32:31 -07:00
Andrej Karpathy	6434009298	attempt to remove PATH_SEP from Makefile	2024-06-03 15:30:45 +00:00
Andrej Karpathy	41dc65a0ef	attempt to move cudnn attention to the llmc directory	2024-06-03 03:23:04 +00:00
Andrej Karpathy	08fc45b492	delete copy paste code for gpt2 model init	2024-06-03 02:08:29 +00:00
Andrej Karpathy	c827dd2f8e	Merge branch 'ngc92-mfu'	2024-06-03 01:41:31 +00:00
Andrej Karpathy	1ac30591f2	add new file llmc/mfu.h for MFU calculations for different GPUs	2024-06-03 01:41:18 +00:00
Andrej	80e0c3b8d9	Merge pull request #512 from gordicaleksa/refactor_encoder_bwd_kernel ... Remove redundant CPU computation in encoder bwd	2024-06-02 16:43:10 -07:00
Andrej	cb5eff0578	Merge pull request #514 from ngc92/compile-fix ... fix compilation with older nvcc	2024-06-02 16:42:31 -07:00
Andrej	ff98d8bda7	Merge pull request #515 from gordicaleksa/fix_adamw_kernel3_num_threads ... Use local params for num blocks in adamw_kernel3	2024-06-02 16:41:59 -07:00
Andrej	1ce30fdc4a	Merge pull request #516 from gordicaleksa/update_random_doc ... Add clarification on Box-Muller	2024-06-02 16:37:37 -07:00
Andrej	252584954e	Merge pull request #519 from gordicaleksa/mem_leak ... Fix mem leak	2024-06-02 16:34:54 -07:00
Andrej Karpathy	1e893f4e11	Merge branch 'gordicaleksa-fix_zero_stage1'	2024-06-02 23:28:22 +00:00
Andrej Karpathy	4944ac5743	adjust comments	2024-06-02 23:28:13 +00:00
Aleksa Gordic	a8a5a444bc	Add ; - ah c my old friend :)	2024-06-02 23:04:47 +02:00
Aleksa Gordic	9a455edc25	Improve comments	2024-06-02 23:03:41 +02:00
Aleksa Gordic	b4e8bd7f66	Fix zero stage 1 norm computation - WIP	2024-06-02 22:45:26 +02:00
Aleksa Gordic	df0a826ef3	Close file	2024-06-02 21:37:52 +02:00
Aleksa Gordic	30d81cce15	Fix mem leak	2024-06-02 21:28:42 +02:00
Aleksa Gordic	79cbc03ccf	Use local params for num blocks	2024-06-02 15:06:06 +02:00
Aleksa Gordic	ae9c957273	Add clarification on Box-Muller	2024-06-02 15:04:45 +02:00
Erik Schultheis	a902323f36	fix compilation with older nvcc	2024-06-02 12:40:28 +03:00
Aleksa Gordic	290c00a362	Remove redundant CPU computation	2024-06-01 18:27:24 +02:00
Andrej	a26041af12	Merge pull request #495 from ChrisDryden/shared_memory ... Removed unnecesary shared memory due to blockreduce using static defined shared memory	2024-06-01 08:12:21 -07:00
Andrej	9cf8c2f135	Merge pull request #509 from ngc92/inplace ... Inplace gelu backward	2024-06-01 08:10:56 -07:00
Andrej	bd0f036ee5	Merge pull request #511 from gordicaleksa/gordicaleksa_fix_docs ... Update compile cmd in the dev/cuda README	2024-06-01 08:09:31 -07:00
Aleksa Gordic	8eafd40615	Update compile cmd in the dev/cuda README	2024-06-01 16:46:32 +02:00
Erik Schultheis	761fb2cf2b	rename gelu_backward to gelu_backward_inplace, and made it obvious in the code that this is an inplace operation	2024-06-01 14:31:26 +03:00
Erik Schultheis	cc4bed0806	added missing checks	2024-06-01 14:30:40 +03:00
Andrej Karpathy	9c8edaffa7	fix recompute bug	2024-05-31 22:38:03 +00:00
Andrej Karpathy	0bc24c36e6	fix bug in resumption of dataloader, have to reload shard	2024-05-31 22:19:37 +00:00
Andrej Karpathy	1de3bef0ac	fix argparse comment too	2024-05-31 22:06:48 +00:00
Andrej Karpathy	60a1037db9	Merge branch 'ChrisDryden-layernorm_backwards_mem_saving'	2024-05-31 22:04:44 +00:00
Andrej Karpathy	f4874a9f63	make things a bit more consistent, keep recompute 1 as default, and add a comment	2024-05-31 22:04:34 +00:00
Andrej Karpathy	629a24020d	Merge branch 'layernorm_backwards_mem_saving' of https://github.com/ChrisDryden/llm.c into ChrisDryden-layernorm_backwards_mem_saving	2024-05-31 21:47:28 +00:00
Andrej Karpathy	ed37d9261b	fix bug in readme	2024-05-30 21:39:47 +00:00
Andrej Karpathy	b48224cb0c	refactor the README file because it was getting crazy	2024-05-30 21:39:06 +00:00
Andrej Karpathy	0fddf06043	more docs on scripts	2024-05-30 21:06:36 +00:00
Andrej Karpathy	fe36e40f2c	also move pyrun script to scripts/	2024-05-30 20:17:16 +00:00
Andrej	d98d3ed82b	code refactor, introduce llmc dir and scripts dir ... The main file train_gpt2.cu is getting out of control. So here we are.	2024-05-30 13:13:04 -07:00
Andrej Karpathy	22d1607f6d	delete these files, they are now in scripts/ folder and better	2024-05-30 20:11:59 +00:00
Andrej Karpathy	fc40ffc4aa	adjust readme to new scripts/ dir	2024-05-30 20:10:04 +00:00
Andrej Karpathy	a0d5dbbda6	scripts folder	2024-05-30 20:00:07 +00:00
Andrej Karpathy	35aa110244	more refactor	2024-05-30 19:39:29 +00:00
Andrej Karpathy	f8f9e49ba3	Merge branch 'master' into feature/libdir	2024-05-30 19:07:51 +00:00
Andrej Karpathy	5f547601a0	further hide the c pytorch bridge logic	2024-05-30 18:20:29 +00:00
Andrej Karpathy	ca27eb8fa5	bugfix don't train on val batch	2024-05-30 18:18:05 +00:00