29 commits

Author	SHA1	Message	Date
Andrej Karpathy	755458d0e1	fix tokenizer omg	2024-08-13 17:24:15 +00:00
Andrej Karpathy	16635d41a2	attempt to fix PR	2024-08-13 02:13:14 +00:00
Aleksa Gordic	68624cc6ac	Refactor tokenize, delegate instead	2024-08-12 20:26:21 +02:00
Aleksa Gordic	e8739c89a0	Refactor magic/version header dict	2024-08-12 20:17:53 +02:00
Aleksa Gordic	eb1b359dae	Replace fire with argparse	2024-08-10 22:45:46 +02:00
Aleksa Gordic	50765638c0	Add fineweb llama 3 support	2024-08-10 22:35:46 +02:00
Aleksa Gordic	f8c1ff590f	Use fire module, update readme	2024-08-10 22:29:19 +02:00
Aleksa Gordic	15b76bf0fb	Add tiny shakespeare	2024-08-10 22:23:33 +02:00
Aleksa Gordic	71419cbd61	Update header description	2024-08-10 21:47:09 +02:00
Aleksa Gordic	d3151c4ace	Add llama 3 tokenization and data loader	2024-08-10 21:25:57 +02:00
Andrej Karpathy	48fee049cb	add edu fineweb sh download script, change paths to my HF repo, add hellaswag to starter pack	2024-07-01 21:41:15 +00:00
Andrej Karpathy	ea36d04e39	small cosmetic fixes to fineweb data file	2024-06-30 15:52:51 +00:00
Andrej Karpathy	ff7a90bd8d	Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_support	2024-06-28 19:35:44 +00:00
Andrej Karpathy	2677d82dcb	add shell script that downloads the .bin files of FineWeb100B directly	2024-06-20 22:39:09 +00:00
“eliebak”	67379a28cc	add edu fineweb support, with 10B and 1000B version	2024-06-02 14:18:44 +00:00
Andrej Karpathy	d0c014372c	add ability to export fineweb100B as well	2024-05-25 17:21:51 +00:00
Andrej Karpathy	08fe3eef43	change default shard size for fineweb10B	2024-05-24 15:56:05 +00:00
Erik Schultheis	b66eb66ac7	update fine-web preprocessing: faster, and much less RAM consumption	2024-05-24 17:45:26 +03:00
Andrej Karpathy	da59861d38	make hellaswag optional eval yay	2024-05-22 22:42:37 +00:00
Andrej Karpathy	67239d9b8f	fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready	2024-05-22 22:21:53 +00:00
Andrej Karpathy	051f3ca53c	first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch	2024-05-22 19:24:20 +00:00
Andrej Karpathy	edb0df967a	continued changes for sharded dataloader	2024-05-22 11:59:09 +00:00
Andrej Karpathy	31310282e1	extend dataloader to be sharded	2024-05-21 21:33:24 +00:00
Andrej Karpathy	d53608820a	the write_datafile function accepts python list, which i think is faster but i didn't check	2024-05-21 17:55:56 +00:00
Andrej Karpathy	1defbd4a19	adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead	2024-05-21 17:05:27 +00:00
Andrej Karpathy	4bbd01fe12	add fineweb, and add the first version of a new write_shard function that contains a header properly	2024-05-21 00:16:07 +00:00
Andrej Karpathy	7d11b7996c	i also forgot to include the readme file for the new dev/data dir	2024-05-20 23:13:42 +00:00
Andrej Karpathy	71774b3c3d	oops forgot to include data_common.py	2024-05-20 23:10:49 +00:00
Andrej Karpathy	722e5b2fe5	refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale	2024-05-20 22:43:40 +00:00