1286 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Andrej Karpathy	11e2fd1cf2	separate out write_checkpoint and delete_checkpoint	2024-06-25 20:46:03 +00:00
Andrej Karpathy	8952f0235b	rolling checkpoints - checkpoints are either MINOR or MAJOR and minor checkpoints get deleted with a rolling window. This is an optimization that will allow us to save state more often, but preserve disk space overall. And if we ever see loss spikes and such, it is easier to reset the state to an earlier checkpoint that is more recent	2024-06-24 23:39:18 +00:00
Andrej	76b133d3f4	Merge pull request #631 from gordicaleksa/minor_refactor2 ... Minor refactor	2024-06-24 14:38:27 -07:00
Andrej	69b50adcf5	Merge pull request #632 from gordicaleksa/multi_node_my ... MPI/TCP/FS for NCCL-init	2024-06-24 14:36:30 -07:00
Aleksa Gordic	4af6a6ab3d	Remove /tmp as default init value for fs_path	2024-06-24 21:26:55 +00:00
Aleksa Gordic	2667aa1e3e	Fix windows include issue	2024-06-24 23:18:38 +02:00
Aleksa Gordic	ac1e37b659	Remove linux socket funcs when windows	2024-06-24 18:56:48 +00:00
Aleksa Gordic	60c95c82aa	Add a multi node readme section	2024-06-24 18:50:56 +00:00
Aleksa Gordic	f9d0e8b67a	Add socket logic for Windows	2024-06-24 18:26:53 +00:00
Aleksa Gordic	05ef1e7dc0	Fix makefile	2024-06-24 14:53:58 +00:00
Aleksa Gordic	c420a522b2	Fix arg permutation	2024-06-24 14:38:49 +00:00
Aleksa Gordic	fbf4f59c8f	Add MPI under multi gpu guard	2024-06-24 16:26:12 +02:00
Aleksa Gordic	a259eeabe5	Fix CI errors	2024-06-24 16:26:12 +02:00
Aleksa Gordic	3abdf9a3e4	Add gpus per node arg	2024-06-24 16:26:12 +02:00
Aleksa Gordic	75a2c8b10f	Add usage for new args	2024-06-24 16:26:12 +02:00
Aleksa Gordic	bfdea887eb	Add launch scripts - tested	2024-06-24 16:26:12 +02:00
Aleksa Gordic	1bc3a5ef6e	Add all 3 init methods	2024-06-24 16:26:12 +02:00
Aleksa Gordic	1c6c5ff0b7	TCP working	2024-06-24 16:26:12 +02:00
Aleksa Gordic	5500b61da5	fs working	2024-06-24 16:26:12 +02:00
Aleksa Gordic	08c055cd72	Free up unified buffer mem	2024-06-24 16:26:12 +02:00
Aleksa Gordic	1bd737546c	Mini refactor	2024-06-24 16:26:12 +02:00
Aleksa Gordic	ac72875c24	Use NCCL stream	2024-06-24 16:26:12 +02:00
Aleksa Gordic	52db158367	Can not call set device after NCCL init	2024-06-24 16:26:12 +02:00
Aleksa Gordic	0addb503ef	Use NCCL where possible	2024-06-24 16:26:12 +02:00
Andrej Karpathy	2a4be7f349	Merge branch 'ngc92-streams-io'	2024-06-23 01:14:50 +00:00
Andrej Karpathy	98e928f1d8	minor fixes for disk device io	2024-06-23 01:14:42 +00:00
Andrej Karpathy	2543b62b88	Merge branch 'streams-io' of https://github.com/ngc92/llm.c into ngc92-streams-io	2024-06-23 00:48:05 +00:00
Aleksa Gordic	71f10e7960	Rename gpt2 backward	2024-06-22 14:32:51 +02:00
Andrej Karpathy	72a2158812	remove a spurious print that is making the table a bit uglier	2024-06-21 15:08:30 +00:00
Andrej Karpathy	c6e21c78b5	Merge branch 'master' of github.com:karpathy/llm.c	2024-06-21 02:40:19 +00:00
Andrej Karpathy	61ed938dfb	delete spurious check it's fine	2024-06-21 02:40:11 +00:00
Andrej	5b82822462	Merge pull request #627 from karpathy/feature/lr_schedulers ... feature/lr_schedulers	2024-06-20 19:31:42 -07:00
Andrej Karpathy	4b5fecc8b2	small touchups and add the wsd schedule	2024-06-21 00:49:01 +00:00
Andrej Karpathy	96ebb16298	simplify lr schedulers	2024-06-21 00:29:47 +00:00
Andrej Karpathy	9abd9ec8a8	Merge branch 'add_lr_schedules' of https://github.com/gordicaleksa/llm.c into gordicaleksa-add_lr_schedules	2024-06-21 00:12:15 +00:00
Andrej	dad58b6800	Merge pull request #625 from gordicaleksa/add_nccl_to_readme ... Add NCCL instruction to README	2024-06-20 16:06:31 -07:00
Andrej Karpathy	2677d82dcb	add shell script that downloads the .bin files of FineWeb100B directly	2024-06-20 22:39:09 +00:00
Aleksa Gordic	5b4d11ac1d	Add NCCL instruction	2024-06-21 00:16:17 +02:00
Aleksa Gordic	716808d9b9	Use C not C++ string header	2024-06-20 16:09:57 +02:00
Aleksa Gordic	3e9fb6cb36	Fix compile issues - add comments	2024-06-20 16:09:57 +02:00
Aleksa Gordic	7416ee425f	Simplify logic since they all share similar args	2024-06-20 16:09:57 +02:00
Aleksa Gordic	9d141ddab9	Add ugly switching between schedulers	2024-06-20 16:09:57 +02:00
Aleksa Gordic	7308c2831c	Add linear LR scheduler	2024-06-20 16:09:57 +02:00
Aleksa Gordic	6f3d099e88	Add constant lr scheduler	2024-06-20 16:09:57 +02:00
Aleksa Gordic	4a1c5a9c0f	Add cyclic triangular LR scheduler	2024-06-20 16:09:57 +02:00
Aleksa Gordic	2eaea3b397	Add schedulers header	2024-06-20 16:09:57 +02:00
Aleksa Gordic	8249736233	Add learning rate schedulers	2024-06-20 16:09:57 +02:00
Andrej	2ddf128365	Merge pull request #615 from gordicaleksa/relax_grad_tests ... Relax grad tensor thresholds in tests	2024-06-19 09:37:42 -07:00
Aleksa Gordic	8079bd9dbb	Relax grad tensor thresholds a bit	2024-06-19 10:21:59 +02:00
Andrej Karpathy	fd564f7b8a	add the gpt2 1558M repro script too	2024-06-19 02:26:39 +00:00