aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-27 20:25:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1515 commits 52 branches 0 tags 21 MiB
Commit graph

1515 commits

This branch
This branch
All branches
Author SHA1 Message Date
Andrej Karpathy
18298f3a40 i misspelled reduced 2024-08-16 18:16:46 +00:00
Andrej Karpathy
8c586f9158 reduce across GPUs nicer 2024-08-16 18:08:34 +00:00
Andrej Karpathy
e6856bc566 fallback to memory allocation of m,v,master_weights on host automatically in case of OOM. will run slower but won't OOM 2024-08-16 17:23:42 +00:00
Andrej Karpathy
2882ec6b9e fix makefile for multigpu setups 2024-08-16 17:22:30 +00:00
Erik Schultheis
f72c1f2c64 hint to host 2024-08-16 01:44:08 +03:00
Erik Schultheis
c8457572fa just try to allocate on device; fallback if that fails 2024-08-16 01:43:56 +03:00
Erik Schultheis
0d52d2a3d7 fall back to cudaMallocManaged for optimizer states if we're out of memory 2024-08-16 01:42:30 +03:00
Andrej
4c84bc743c
Merge pull request #740 from karpathy/gordicaleksa-fix_dataloader2 
Gordicaleksa fix dataloader2
 2024-08-13 11:18:57 -07:00
Andrej Karpathy
755458d0e1 fix tokenizer omg 2024-08-13 17:24:15 +00:00
Andrej Karpathy
16635d41a2 attempt to fix PR 2024-08-13 02:13:14 +00:00
Andrej Karpathy
9740a652cd Merge branch 'fix_dataloader2' of https://github.com/gordicaleksa/llm.c into gordicaleksa-fix_dataloader2 2024-08-12 23:50:10 +00:00
Andrej
1787210306
Merge pull request #738 from ademeure/faster_compile 
Improve compile time (simple makefile changes)
 2024-08-12 16:45:15 -07:00
Aleksa Gordic
68624cc6ac Refactor tokenize, delegate instead 2024-08-12 20:26:21 +02:00
Aleksa Gordic
e8739c89a0 Refactor magic/version header dict 2024-08-12 20:17:53 +02:00
ademeure
fc8834409b Improve compile time via nvidia-smi instead of __nvcc_device_query (& add "--threads=0"). 2024-08-12 17:07:36 +00:00
ademeure
9b1c14bd93 Improve compile time via nvidia-smi instead of __nvcc_device_query (& add "--threads=0"). 2024-08-12 16:51:39 +00:00
Aleksa Gordic
eb1b359dae Replace fire with argparse 2024-08-10 22:45:46 +02:00
Aleksa Gordic
311c149a2a Add fire to reqs 2024-08-10 22:39:07 +02:00
Aleksa Gordic
50765638c0 Add fineweb llama 3 support 2024-08-10 22:35:46 +02:00
Aleksa Gordic
f8c1ff590f Use fire module, update readme 2024-08-10 22:29:19 +02:00
Aleksa Gordic
15b76bf0fb Add tiny shakespeare 2024-08-10 22:23:33 +02:00
Aleksa Gordic
b018b0c85c Change default dataset path to tiny stories 2024-08-10 21:50:39 +02:00
Aleksa Gordic
71419cbd61 Update header description 2024-08-10 21:47:09 +02:00
Aleksa Gordic
d3151c4ace Add llama 3 tokenization and data loader 2024-08-10 21:25:57 +02:00
Andrej
6e6a528111
Merge pull request #725 from gordicaleksa/llama 
Add LLaMA 3 Python support
 2024-08-08 14:08:20 -07:00
Aleksa Gordic
d773c88e49 Remove unnecessary comment 2024-08-08 21:06:01 +02:00
Aleksa Gordic
c5c87fc4e7 Add comments, fix stop tokens 2024-08-08 20:54:17 +02:00
Aleksa Gordic
8b672ffcb9 Add data loader not implemented exception 2024-08-08 18:55:17 +02:00
Aleksa Gordic
f1c91f8ae3 Address Andrej's PR comments 2024-08-08 18:45:04 +02:00
Aleksa Gordic
89addd3e3e Remove pycache 2024-08-08 18:20:51 +02:00
Aleksa Gordic
ac01536b99 Remove the whole llmc_py 2024-08-08 18:20:18 +02:00
Aleksa Gordic
dfd459bfc4 Remove prompts.json 2024-08-08 18:19:40 +02:00
Aleksa Gordic
624ed3ce30 Rename the new file to train llama3 2024-08-08 18:15:37 +02:00
Aleksa Gordic
b7c98c9365 Add llama config error check 2024-08-08 18:12:35 +02:00
Aleksa Gordic
b25e325c69 Add explicit external mask 2024-08-08 17:55:01 +02:00
Aleksa Gordic
d4ef9c5afd Remove llmc_py, single file 2024-08-08 17:09:41 +02:00
Aleksa Gordic
72dcfeb404 Remove float(-inf) 2024-08-04 22:20:46 +02:00
Aleksa Gordic
8e55d16863 Support HF & Meta models 2024-08-04 16:27:12 +02:00
Aleksa Gordic
0749a4af42 Minor refactor - remove wpe pos array from fwd 2024-08-04 09:28:17 +00:00
Aleksa Gordic
4b386a2a89 Test training loop works 2024-08-04 09:13:29 +00:00
Aleksa Gordic
cd902735b2 Fix saving model / state logic 2024-08-04 08:54:21 +00:00
Aleksa Gordic
7768a36f37 Add prompts file 2024-08-04 10:07:48 +02:00
Aleksa Gordic
879cc5f46c Remove init weights as it's gpt-2 specific 2024-08-04 08:00:44 +00:00
Aleksa Gordic
bad7857dff Done - need to test train loop and saving model 2024-08-03 22:37:54 +02:00
Aleksa Gordic
b1acb5988f Got to llama 3 inference (end) 2024-08-03 22:22:38 +02:00
Aleksa Gordic
d855c9695a Refactoring - got to main 2024-08-03 22:16:13 +02:00
Aleksa Gordic
c1d2b7fec1 Removed from pretrained for GPT-2 2024-08-03 22:02:02 +02:00
Aleksa Gordic
8a48df7b16 MLP GPT2->LLaMA3 2024-08-03 21:53:26 +02:00
Aleksa Gordic
ccdbdfd4ce Going towards pure llama 3 file - fixed attn 2024-08-03 21:49:45 +02:00
Aleksa Gordic
8919b66c37 Fix generate for gpt-2 2024-08-03 14:14:05 +02:00