7 commits

Author	SHA1	Message	Date
Andrej Karpathy	d0c014372c	add ability to export fineweb100B as well	2024-05-25 17:21:51 +00:00
Andrej Karpathy	08fe3eef43	change default shard size for fineweb10B	2024-05-24 15:56:05 +00:00
Erik Schultheis	b66eb66ac7	update fine-web preprocessing: faster, and much less RAM consumption	2024-05-24 17:45:26 +03:00
Andrej Karpathy	edb0df967a	continued changes for sharded dataloader	2024-05-22 11:59:09 +00:00
Andrej Karpathy	31310282e1	extend dataloader to be sharded	2024-05-21 21:33:24 +00:00
Andrej Karpathy	1defbd4a19	adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead	2024-05-21 17:05:27 +00:00
Andrej Karpathy	4bbd01fe12	add fineweb, and add the first version of a new write_shard function that contains a header properly	2024-05-21 00:16:07 +00:00