3 commits

Author	SHA1	Message	Date
Andrej Karpathy	d53608820a	the write_datafile function accepts python list, which i think is faster but i didn't check	2024-05-21 17:55:56 +00:00
Andrej Karpathy	1defbd4a19	adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead	2024-05-21 17:05:27 +00:00
Andrej Karpathy	722e5b2fe5	refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale	2024-05-20 22:43:40 +00:00

Renamed from prepro_tinystories.py (Browse further)