9 commits

Author	SHA1	Message	Date
Andrej Karpathy	16635d41a2	attempt to fix PR	2024-08-13 02:13:14 +00:00
Aleksa Gordic	e8739c89a0	Refactor magic/version header dict	2024-08-12 20:17:53 +02:00
Aleksa Gordic	d3151c4ace	Add llama 3 tokenization and data loader	2024-08-10 21:25:57 +02:00
Erik Schultheis	b66eb66ac7	update fine-web preprocessing: faster, and much less RAM consumption	2024-05-24 17:45:26 +03:00
Andrej Karpathy	051f3ca53c	first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch	2024-05-22 19:24:20 +00:00
Andrej Karpathy	d53608820a	the write_datafile function accepts python list, which i think is faster but i didn't check	2024-05-21 17:55:56 +00:00
Andrej Karpathy	1defbd4a19	adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead	2024-05-21 17:05:27 +00:00
Andrej Karpathy	4bbd01fe12	add fineweb, and add the first version of a new write_shard function that contains a header properly	2024-05-21 00:16:07 +00:00
Andrej Karpathy	71774b3c3d	oops forgot to include data_common.py	2024-05-20 23:10:49 +00:00