14 commits

Author	SHA1	Message	Date
Andrej Karpathy	755458d0e1	fix tokenizer omg	2024-08-13 17:24:15 +00:00
Andrej Karpathy	16635d41a2	attempt to fix PR	2024-08-13 02:13:14 +00:00
Aleksa Gordic	68624cc6ac	Refactor tokenize, delegate instead	2024-08-12 20:26:21 +02:00
Aleksa Gordic	50765638c0	Add fineweb llama 3 support	2024-08-10 22:35:46 +02:00
Aleksa Gordic	d3151c4ace	Add llama 3 tokenization and data loader	2024-08-10 21:25:57 +02:00
Andrej Karpathy	ea36d04e39	small cosmetic fixes to fineweb data file	2024-06-30 15:52:51 +00:00
“eliebak”	67379a28cc	add edu fineweb support, with 10B and 1000B version	2024-06-02 14:18:44 +00:00
Andrej Karpathy	d0c014372c	add ability to export fineweb100B as well	2024-05-25 17:21:51 +00:00
Andrej Karpathy	08fe3eef43	change default shard size for fineweb10B	2024-05-24 15:56:05 +00:00
Erik Schultheis	b66eb66ac7	update fine-web preprocessing: faster, and much less RAM consumption	2024-05-24 17:45:26 +03:00
Andrej Karpathy	edb0df967a	continued changes for sharded dataloader	2024-05-22 11:59:09 +00:00
Andrej Karpathy	31310282e1	extend dataloader to be sharded	2024-05-21 21:33:24 +00:00
Andrej Karpathy	1defbd4a19	adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead	2024-05-21 17:05:27 +00:00
Andrej Karpathy	4bbd01fe12	add fineweb, and add the first version of a new write_shard function that contains a header properly	2024-05-21 00:16:07 +00:00