Erik Schultheis
67502c7f8e
moved device utilities into a separate file, and follow-up cleanup
2024-06-03 23:58:47 +03:00
Andrej Karpathy
020b15ad77
small more fixes
2024-06-03 19:32:32 +00:00
Andrej Karpathy
6434009298
attempt to remove PATH_SEP from Makefile
2024-06-03 15:30:45 +00:00
Andrej Karpathy
41dc65a0ef
attempt to move cudnn attention to the llmc directory
2024-06-03 03:23:04 +00:00
Jun Zhang
9e645314e6
Check if file exists using platform specific commands & Add *.o to gitignore
...
Signed-off-by: Jun Zhang <jun@junz.org>
2024-05-20 00:00:21 +08:00
Marco van Zwetselaar
6da5e63e2c
Fix detection of cudnn-frontend in '.' on Linux
...
Plus minor legibility fixes in the cudnn-frontend explanation.
2024-05-10 10:57:39 +03:00
Joe Halabi
69aa64cc80
Adds cuDNN install instructions for a smaller but sufficient package
...
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).
- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)
- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions
Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com>
2024-05-08 14:53:31 -07:00
Anerudhan
0ce5fcf7e8
Rename cudnn_att.cu to cudnn_att.cpp to speed up compilation.
2024-05-08 19:53:48 +00:00
Ross Wheeler
134f4c7fc8
Adding two directory search for cuDNN frontend files
...
Search in $(HOME) and in the current directory.
2024-05-06 12:38:40 -07:00
rosslwheeler
dedd86c1b3
Merge branch 'karpathy:master' into cudnn_makefile_changes
2024-05-06 11:20:22 -07:00
Ross Wheeler
0c4908d815
Adding CI check to disable auto-detect
2024-05-06 01:11:27 -07:00
Ross Wheeler
2bfd2b6a7e
Auto-detect GPU capability
...
3 cases tested on Windows 11 and Ubuntu 22.04
2024-05-06 00:46:28 -07:00
Ross Wheeler
2d4e5fd840
Minor fixes for Makefile for cudnn and windows support
...
Tested on Ubuntu Linux 22.04 and Windows 11
2024-05-04 23:26:57 -07:00
Erik Schultheis
b087b9c819
don't compile/link cudnn if not asked for it
2024-05-05 02:54:42 +03:00
Erik Schultheis
aa5bb258b6
fixed up test and profile targets
2024-05-04 11:59:59 +03:00
Erik Schultheis
67a82a611e
first attempt at moving cudnn out of the main file for faster compiles
2024-05-03 23:08:44 +03:00
Andrej Karpathy
e780b56974
small fixes for profiling
2024-05-03 15:54:29 +00:00
Andrej Karpathy
69d40ebfff
add readme and search for cudd in home dir
2024-05-01 17:43:50 +00:00
Andrej Karpathy
35640651b3
updates to makefile: docs, override, etc
2024-05-01 17:32:52 +00:00
Andrej Karpathy
04718fbbd1
round 1 of small fixes for cudnn support
2024-05-01 17:07:10 +00:00
ademeure
7beff72f5f
Add makefile changes for cuDNN
2024-05-01 15:31:30 +01:00
Andrej Karpathy
906d22f5b6
fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags
2024-04-29 23:27:55 +00:00
Ross Wheeler
fcef97d6a1
Adding back PFLAGS
...
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
2024-04-29 08:53:50 -07:00
Ross Wheeler
c9ab8ee123
Updating Makefile with latest changes
...
Add check with Windows OS
2024-04-29 03:00:46 -07:00
Andrej
18b41b4df6
Merge pull request #279 from Ricardicus/prec-ci
...
Updating the CI to build different precisions
2024-04-28 13:47:48 -07:00
Andrej Karpathy
49228b091a
add small comment on -t=0
2024-04-28 20:20:51 +00:00
Rickard Hallerbäck
b522333851
Updating the CI to build different precisions
2024-04-28 22:17:47 +02:00
Andrej
10aa24e2cc
Merge pull request #269 from ChrisDryden/patch-3
...
Enable multithreading in nvcc
2024-04-28 13:16:47 -07:00
Andrej Karpathy
9d70d9a1b2
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
2024-04-28 16:17:22 +00:00
Andrej Karpathy
4f7d8d9f44
allow user to make different precisions, add prints and error handling around precisions
2024-04-28 16:08:27 +00:00
Chris Dryden
2954d90f30
Enable multithreading in nvcc
...
Tested locally and reduced compilation time by 200ms, unfortunately for me upgrading to 12.4 made my compilations times slow by 2x but at least this can make it a bit faster
2024-04-27 14:14:02 -07:00
Andrej Karpathy
a00a3a88ef
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
2024-04-26 18:11:09 +00:00
Andrej Karpathy
2e17140d33
more defensive check for openmpi install, and an install command
2024-04-26 16:27:57 +00:00
Petr Zhizhin
d49e8301eb
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
2024-04-24 22:11:00 +00:00
Andrej Karpathy
ed6387a695
updates to readme, and introduce the test fp32 cuda file too
2024-04-23 17:41:42 +00:00
Andrej Karpathy
095d27662b
checkpoint the fp32 CUDA implementation to separate file. our mainline iteration will now continue in the new (mixed precision) file
2024-04-23 17:21:44 +00:00
Rickard Hallerbäck
c7aad65a04
Adding a workflow that builds with and without CUDA and OMP and on CPU also tests on ubuntu and macos
2024-04-22 02:14:09 +02:00
Andrej Karpathy
fced6d180f
small adjustments to Makefile and gitignore
2024-04-21 17:40:26 +00:00
Rickard Hallerbäck
3cd6f83e32
removing some warnings
2024-04-20 22:03:35 +02:00
Rickard Hallerbäck
be106f09a4
Merge branch 'master' into master
2024-04-20 22:00:02 +02:00
Coenraad
f534e4bdfe
Fixes -Ofast optimizations breaking model by skipping them for gelu_backward
2024-04-20 21:33:00 +02:00
Rickard Hallerbäck
280023bf8e
Support for vanilla build without CUDA tools
2024-04-20 21:32:35 +02:00
Erik Schultheis
9c1c1f0cec
helper target that gets compiled with lineinfo and runs a single layer forward-backward pass
2024-04-20 03:19:23 +03:00
ent0n29
66c30fb55f
-fno-finite-math-only for almost 2x speed up
2024-04-16 12:49:54 +02:00
Coenraad Loubser
752cbe068d
Update Makefile with -march=native
...
-march=native Results in a 30% speedup on all the platforms I've tried, for train_gpt2 (Admittedly, all older Intel and AMD ones)
```
model name : Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz
step 0: train loss 5.356172 (took 15197.876634 ms)
vs
step 0: train loss 5.356185 (took 10418.548668 ms)
model name : AMD Ryzen 5 3600 6-Core Processor
step 0: train loss 5.356173 (took 3599.359234 ms)
vs
step 0: train loss 5.356185 (took 2708.045790 ms)
```
2024-04-15 19:52:19 +02:00
Andrej
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
Scott Haleen
47b2f9312c
fixed homebrew path for intel based macs
2024-04-10 23:01:07 -04:00
Andrej Karpathy
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
Rickard Hallerbäck
1df7bceff9
using the compiler at hand
2024-04-09 06:22:03 +02:00
Rickard Hallerbäck
b2228d0e83
Minor correction for openmp check
2024-04-08 22:33:47 +02:00
