10 commits

Author	SHA1	Message	Date
Andrej Karpathy	755458d0e1	fix tokenizer omg	2024-08-13 17:24:15 +00:00
Andrej Karpathy	16635d41a2	attempt to fix PR	2024-08-13 02:13:14 +00:00
Aleksa Gordic	eb1b359dae	Replace fire with argparse	2024-08-10 22:45:46 +02:00
Aleksa Gordic	f8c1ff590f	Use fire module, update readme	2024-08-10 22:29:19 +02:00
Aleksa Gordic	15b76bf0fb	Add tiny shakespeare	2024-08-10 22:23:33 +02:00
Aleksa Gordic	71419cbd61	Update header description	2024-08-10 21:47:09 +02:00
Aleksa Gordic	d3151c4ace	Add llama 3 tokenization and data loader	2024-08-10 21:25:57 +02:00
Andrej Karpathy	d53608820a	the write_datafile function accepts python list, which i think is faster but i didn't check	2024-05-21 17:55:56 +00:00
Andrej Karpathy	1defbd4a19	adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead	2024-05-21 17:05:27 +00:00
Andrej Karpathy	722e5b2fe5	refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale	2024-05-20 22:43:40 +00:00

Renamed from prepro_tinystories.py (Browse further)