4 commits

Author	SHA1	Message	Date
Andrej Karpathy	edb0df967a	continued changes for sharded dataloader	2024-05-22 11:59:09 +00:00
Andrej Karpathy	31310282e1	extend dataloader to be sharded	2024-05-21 21:33:24 +00:00
Andrej Karpathy	1defbd4a19	adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead	2024-05-21 17:05:27 +00:00
Andrej Karpathy	4bbd01fe12	add fineweb, and add the first version of a new write_shard function that contains a header properly	2024-05-21 00:16:07 +00:00