Andrej
68535cb6ce
Merge pull request #649 from karpathy/feature/openmpi_flexible_path
...
tune openmpi make a bit more convenient
2024-06-26 09:11:22 -07:00
Andrej Karpathy
7c283df9c4
use ?= for OPENMPI_DIR so it can be overriden
2024-06-26 16:06:54 +00:00
Andrej Karpathy
2a8f82f14f
tune openmpi make a bit more convenient
2024-06-26 15:39:10 +00:00
Rickard Hallerbäck
66cd8c3f1f
Makefile: respect the NO_MULTI_GPU
2024-06-26 14:55:02 +02:00
chinthysl
934e6f22c5
Fix missing compiler flag
2024-06-25 09:35:24 +00:00
Aleksa Gordic
2667aa1e3e
Fix windows include issue
2024-06-24 23:18:38 +02:00
Aleksa Gordic
f9d0e8b67a
Add socket logic for Windows
2024-06-24 18:26:53 +00:00
Aleksa Gordic
05ef1e7dc0
Fix makefile
2024-06-24 14:53:58 +00:00
Aleksa Gordic
c420a522b2
Fix arg permutation
2024-06-24 14:38:49 +00:00
Aleksa Gordic
a259eeabe5
Fix CI errors
2024-06-24 16:26:12 +02:00
Aleksa Gordic
bfdea887eb
Add launch scripts - tested
2024-06-24 16:26:12 +02:00
Vyom Sharma
c24ef4a4b4
Merge branch 'karpathy:master' into master
2024-06-07 14:20:55 +05:30
Erik Schultheis
43f013631b
set compile flag and add ci check
2024-06-06 10:55:33 +03:00
Vyom Sharma
1e8efe94c0
Update Makefile - revert back
2024-06-05 12:02:09 +05:30
Vyom Sharma
453fde77fd
Merge branch 'karpathy:master' into master
2024-06-04 23:39:10 +05:30
Ross Wheeler
4bfb23948d
Make clean fix and Windows cuDNN build fix
2024-06-03 20:26:52 -07:00
Erik Schultheis
67502c7f8e
moved device utilities into a separate file, and follow-up cleanup
2024-06-03 23:58:47 +03:00
Andrej Karpathy
020b15ad77
small more fixes
2024-06-03 19:32:32 +00:00
Andrej Karpathy
6434009298
attempt to remove PATH_SEP from Makefile
2024-06-03 15:30:45 +00:00
vyom1611
9b540fa859
Makefile updated to match script
2024-06-03 18:59:57 +05:30
Andrej Karpathy
41dc65a0ef
attempt to move cudnn attention to the llmc directory
2024-06-03 03:23:04 +00:00
Jun Zhang
9e645314e6
Check if file exists using platform specific commands & Add *.o to gitignore
...
Signed-off-by: Jun Zhang <jun@junz.org>
2024-05-20 00:00:21 +08:00
Marco van Zwetselaar
6da5e63e2c
Fix detection of cudnn-frontend in '.' on Linux
...
Plus minor legibility fixes in the cudnn-frontend explanation.
2024-05-10 10:57:39 +03:00
Joe Halabi
69aa64cc80
Adds cuDNN install instructions for a smaller but sufficient package
...
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).
- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)
- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions
Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com>
2024-05-08 14:53:31 -07:00
Anerudhan
0ce5fcf7e8
Rename cudnn_att.cu to cudnn_att.cpp to speed up compilation.
2024-05-08 19:53:48 +00:00
Ross Wheeler
134f4c7fc8
Adding two directory search for cuDNN frontend files
...
Search in $(HOME) and in the current directory.
2024-05-06 12:38:40 -07:00
rosslwheeler
dedd86c1b3
Merge branch 'karpathy:master' into cudnn_makefile_changes
2024-05-06 11:20:22 -07:00
Ross Wheeler
0c4908d815
Adding CI check to disable auto-detect
2024-05-06 01:11:27 -07:00
Ross Wheeler
2bfd2b6a7e
Auto-detect GPU capability
...
3 cases tested on Windows 11 and Ubuntu 22.04
2024-05-06 00:46:28 -07:00
Ross Wheeler
2d4e5fd840
Minor fixes for Makefile for cudnn and windows support
...
Tested on Ubuntu Linux 22.04 and Windows 11
2024-05-04 23:26:57 -07:00
Erik Schultheis
b087b9c819
don't compile/link cudnn if not asked for it
2024-05-05 02:54:42 +03:00
Erik Schultheis
aa5bb258b6
fixed up test and profile targets
2024-05-04 11:59:59 +03:00
Erik Schultheis
67a82a611e
first attempt at moving cudnn out of the main file for faster compiles
2024-05-03 23:08:44 +03:00
Andrej Karpathy
e780b56974
small fixes for profiling
2024-05-03 15:54:29 +00:00
Andrej Karpathy
69d40ebfff
add readme and search for cudd in home dir
2024-05-01 17:43:50 +00:00
Andrej Karpathy
35640651b3
updates to makefile: docs, override, etc
2024-05-01 17:32:52 +00:00
Andrej Karpathy
04718fbbd1
round 1 of small fixes for cudnn support
2024-05-01 17:07:10 +00:00
ademeure
7beff72f5f
Add makefile changes for cuDNN
2024-05-01 15:31:30 +01:00
Andrej Karpathy
906d22f5b6
fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags
2024-04-29 23:27:55 +00:00
Ross Wheeler
fcef97d6a1
Adding back PFLAGS
...
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
2024-04-29 08:53:50 -07:00
Ross Wheeler
c9ab8ee123
Updating Makefile with latest changes
...
Add check with Windows OS
2024-04-29 03:00:46 -07:00
Andrej
18b41b4df6
Merge pull request #279 from Ricardicus/prec-ci
...
Updating the CI to build different precisions
2024-04-28 13:47:48 -07:00
Andrej Karpathy
49228b091a
add small comment on -t=0
2024-04-28 20:20:51 +00:00
Rickard Hallerbäck
b522333851
Updating the CI to build different precisions
2024-04-28 22:17:47 +02:00
Andrej
10aa24e2cc
Merge pull request #269 from ChrisDryden/patch-3
...
Enable multithreading in nvcc
2024-04-28 13:16:47 -07:00
Andrej Karpathy
9d70d9a1b2
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
2024-04-28 16:17:22 +00:00
Andrej Karpathy
4f7d8d9f44
allow user to make different precisions, add prints and error handling around precisions
2024-04-28 16:08:27 +00:00
Chris Dryden
2954d90f30
Enable multithreading in nvcc
...
Tested locally and reduced compilation time by 200ms, unfortunately for me upgrading to 12.4 made my compilations times slow by 2x but at least this can make it a bit faster
2024-04-27 14:14:02 -07:00
Andrej Karpathy
a00a3a88ef
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
2024-04-26 18:11:09 +00:00
Andrej Karpathy
2e17140d33
more defensive check for openmpi install, and an install command
2024-04-26 16:27:57 +00:00
