aparler/c-llm
1
0
Fork 0
mirror of https://github.com/karpathy/llm.c.git synced 2026-07-28 20:35:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1022 commits 52 branches 0 tags 21 MiB
Commit graph

1022 commits

This branch
This branch
All branches
Author SHA1 Message Date
ademeure
e1f89b3043 remove BF16 default from classified_fused before PR 2024-05-07 00:57:01 +01:00
ademeure
1ea7f9bf25 tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changes 2024-05-07 00:54:50 +01:00
Andrej
ec9fd51d5c
Merge pull request #375 from Chillee/patch-1 
Don't return logits during training for PyTorch baseline
 2024-05-07 01:49:02 +02:00
Andrej
aa54a0a808
Merge pull request #377 from ademeure/profiler_efficiency 
Add DRAM BW, Tensor Core Utilisation, & Efficiency Stats
 2024-05-07 01:45:33 +02:00
ademeure
9b55ea8cf5 More crazy optimisations to layernorm_backward, fused_classifier, and matmul_backward_bias. 2024-05-07 00:11:42 +01:00
Ross Wheeler
f7d7760084 Non-standard C syntax replace with standard C 
Also, adding unistd.h back because that has some changes required to find M_PI.
 2024-05-06 13:56:17 -07:00
ademeure
5adb6ef2c1 Add tensor core and overall efficiency stats to profiler script. 2024-05-06 21:24:42 +01:00
Horace He
69f6c4f765
Don't return logits during training for PyTorch baseline 
This improves perf somewhat, since currently it's always returning logits (which thus need to be materialized).
 2024-05-06 13:10:17 -07:00
Ross Wheeler
134f4c7fc8 Adding two directory search for cuDNN frontend files 
Search in $(HOME) and in the current directory.
 2024-05-06 12:38:40 -07:00
rosslwheeler
dedd86c1b3
Merge branch 'karpathy:master' into cudnn_makefile_changes 2024-05-06 11:20:22 -07:00
Andrej
4274d95b4a
Merge pull request #365 from lancerts/classifier-fix 
Minor update on the classifier_fused.cu
 2024-05-06 12:04:16 +02:00
Andrej Karpathy
6798dd4b91 Merge branch 'rosslwheeler-Makefile-detect-gpu' 2024-05-06 10:01:57 +00:00
chinthysl
7a8f471bc7 further reorganization 2024-05-06 17:49:58 +08:00
Ross Wheeler
5344cc30b8 Merge branch 'Makefile-detect-gpu' of https://github.com/rosslwheeler/llm.c into Makefile-detect-gpu 2024-05-06 01:11:37 -07:00
Ross Wheeler
0c4908d815 Adding CI check to disable auto-detect 2024-05-06 01:11:27 -07:00
rosslwheeler
98aa83b282
Merge branch 'karpathy:master' into Makefile-detect-gpu 2024-05-06 00:58:09 -07:00
Ross Wheeler
2bfd2b6a7e Auto-detect GPU capability 
3 cases tested on Windows 11 and Ubuntu 22.04
 2024-05-06 00:46:28 -07:00
Chinthaka
aae7145495
Merge branch 'master' into zero-stage1 2024-05-06 10:36:24 +08:00
lancer
bbfe8c989c Minor update on the code 2024-05-05 16:59:58 -07:00
Andrej
9e4f9df9a2
Merge pull request #337 from lancerts/make-fix 
Makefile type fix: move gelu_backward to backward block
 2024-05-05 15:37:02 -07:00
Andrej
a7c1e4437d
Merge pull request #364 from ngc92/cleanup_may4-nvcc11 
make things compile with nvcc11
 2024-05-05 14:42:49 -07:00
Andrej Karpathy
d6e8f5e026 Merge branch 'ademeure-cleanup_may4' 2024-05-05 21:29:41 +00:00
Erik Schultheis
804a9af1dc make things compile with nvcc11 2024-05-05 23:43:02 +03:00
ademeure
876ab93c0a Add FP16 path for atomicStochasticAdd (+remove __bfloat1622float2 to work on older CUDAs) + fixes 2024-05-05 15:05:59 +01:00
ademeure
9910a40863 Removed makefile change so we can integrate #339 instead which feels slightly cleaner 2024-05-05 14:42:32 +01:00
Ross Wheeler
42176ad4d5 Merge branch 'cudnn_makefile_changes' of https://github.com/rosslwheeler/llm.c into cudnn_makefile_changes 2024-05-04 23:27:21 -07:00
Ross Wheeler
2d4e5fd840 Minor fixes for Makefile for cudnn and windows support 
Tested on Ubuntu Linux 22.04 and Windows 11
 2024-05-04 23:26:57 -07:00
Ross Wheeler
9663719c62 Minor fixes for Make for cudnn and windows support 
Tested on Ubuntu Linux 22.04 and Windows 11
 2024-05-04 23:24:41 -07:00
ademeure
c15ca1f4cf Fix makefile 2024-05-05 02:49:16 +01:00
ademeure
7789738879 Add PTX back to binary + fix whitespaces 2024-05-05 02:44:24 +01:00
ademeure
8675104b73 Compile for the user's GPU architecture using nvidia-smi query on Linux 2024-05-05 02:24:31 +01:00
ademeure
83ec4b8e70 Slightly reduce lines of code in cudnn_att 2024-05-05 02:09:52 +01:00
ademeure
c261eecda8 Merge remote-tracking branch 'karpathy/master' into cleanup_may4 2024-05-05 01:49:25 +01:00
Andrej Karpathy
64b6c2a77f Merge branch 'ngc92-separate-compilation' 2024-05-05 00:41:18 +00:00
Andrej Karpathy
f2224f2eb2 Merge branch 'separate-compilation' of https://github.com/ngc92/llm.c into ngc92-separate-compilation 2024-05-05 00:32:31 +00:00
Andrej Karpathy
8168b78549 Merge branch 'ademeure-less_idle_more_brrr-3' 2024-05-05 00:27:15 +00:00
Andrej Karpathy
ce333de156 add ema to tok/s 2024-05-05 00:18:39 +00:00
Andrej Karpathy
2642ffd19c Merge branch 'less_idle_more_brrr' of https://github.com/ademeure/llm.c into ademeure-less_idle_more_brrr-3 2024-05-05 00:05:38 +00:00
ademeure
a0b79358bd Fix loss not being calculated for forward-only paths 2024-05-05 01:01:59 +01:00
Andrej
6c179faaeb
Merge pull request #349 from lancerts/fix-include 
Remove the duplicated #include <assert.h>
 2024-05-04 17:01:32 -07:00
Erik Schultheis
b087b9c819 don't compile/link cudnn if not asked for it 2024-05-05 02:54:42 +03:00
ademeure
34304361ba fix merge conflict 2024-05-05 00:16:49 +01:00
ademeure
ec0ab2d2d0 Remove arch=native as it only available on recent CUDA versions 2024-05-05 00:00:58 +01:00
ademeure
18d7ed9242 revert profile.py changes for now 2024-05-04 23:46:39 +01:00
ademeure
abaaceb801 Added makefile gencode changes 2024-05-04 23:43:13 +01:00
ademeure
bfb9c51446 refactoring & remove unused functions to reduce LOC (+wip profile.py improvements) 2024-05-04 23:40:15 +01:00
ademeure
66b9755ce9 Use template for blockReduce's warp reduction + add out_of_bounds parameter 2024-05-04 18:48:54 +01:00
ademeure
8d593b916e fix cuda stream bug for test_gpt2cu... by removing most of the memcpy/memset parallelism for now (keeping loss_event to avoid sync between forward and backward). 2024-05-04 18:18:51 +01:00
Erik Schultheis
19c290d7e6 improved debugging for cudnn 2024-05-04 12:31:15 +03:00
Erik Schultheis
aa5bb258b6 fixed up test and profile targets 2024-05-04 11:59:59 +03:00