From 38fd42d3e38751fef6976e2633eab43d0ec8f2e2 Mon Sep 17 00:00:00 2001 From: Andrej Karpathy Date: Sat, 20 Apr 2024 00:41:13 +0000 Subject: [PATCH] also add a quick start CPU section --- README.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 006e110..489910e 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,20 @@ make train_gpt2cu ./train_gpt2cu ``` -The above lines (1) download the [tinyshakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) dataset, tokenize it with the GPT-2 Tokenizer, (2) save the GPT-2 (124M) weights, (3) init from them in C/CUDA and train for one epoch on tineshakespeare with AdamW (using batch size 4, contenxt length 1024, total of 74 steps), evaluate validation loss, and sample some text. +The above lines (1) download the [tinyshakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) dataset, tokenize it with the GPT-2 Tokenizer, (2) download and save the GPT-2 (124M) weights, (3) init from them in C/CUDA and train for one epoch on tineshakespeare with AdamW (using batch size 4, context length 1024, total of 74 steps), evaluate validation loss, and sample some text. + +## quick start (CPU) + +The "I am so GPU poor that I don't even have one" section. No worries, run: + +```bash +python prepro_tinyshakespeare.py +python train_gpt2.py +make train_gpt2 +OMP_NUM_THREADS=8 ./train_gpt2 +``` + +The above lines (1) download the [tinyshakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) dataset, tokenize it with the GPT-2 Tokenizer, (2) download and save the GPT-2 (124M) weights, (3) init from them in C and train for 40 steps on tineshakespeare with AdamW (using batch size 4, context length only 64), evaluate validation loss, and sample some text. Honestly, unless you have a beefy CPU (and can crank up the number of OMP threads in the launch command), you're not going to get that far on CPU training LLMs, but it might be a good demo/reference. ## train