Commit graph

Select branches

Hide pull requests

ademeure-layernormbf16-3

attempt_fix_zero1

feature/bias_backward2

feature/cublaslt

feature/cudnn

feature/delete_backward_act

feature/finetune_llama31py

feature/fix_int_overflow

feature/fp32_weight_master_copy

feature/gpt3v1

feature/grad_accum

feature/grad_accum3

feature/hellaswag

feature/hellaswagc

feature/hf_eval

feature/libdir

feature/load_bf16

feature/lr_schedulers

feature/managed2

feature/nccl-only

feature/openmpi_flexible_path

feature/organize_data

feature/outliers

feature/overlap-backward-reduce

feature/packed128

feature/permuted_dataloader

feature/pytorch_parity

feature/random_init

feature/recompute

feature/refactor_part2

feature/restore_from_master

feature/resume

feature/rolling_checkpoints

feature/sharded_data

feature/streams

feature/weight_decay_2d_only

feature/wikitext103

feature/write_checkpoints

fix/illegalaccess

fix/reshuffle_zero_grad

gordicaleksa-fix_dataloader2

llama3

master

move/attention

move/encoder_kernel

move/fused_classifier

move/gelu

move/matmul

overfit-batch-actually

refactor/code-to-zerocuh

separate_tokenizer

try/move_cudnn_to_llmc

#100

#101

#107

#108

#109

#111

#112

#113

#115

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#126

#128

#129

#130

#130

#131

#132

#133

#134

#137

#139

#14

#140

#141

#142

#143

#145

#149

#15

#150

#151

#152

#153

#155

#156

#159

#16

#160

#161

#162

#163

#164

#165

#166

#167

#168

#169

#170

#172

#173

#174

#175

#177

#179

#180

#181

#182

#183

#184

#185

#186

#186

#187

#188

#188

#189

#191

#192

#193

#194

#195

#196

#197

#198

#199

#2

#20

#200

#201

#207

#208

#209

#21

#210

#211

#211

#213

#214

#215

#216

#218

#221

#222

#225

#226

#227

#227

#228

#23

#231

#233

#233

#235

#235

#236

#238

#239

#24

#240

#241

#241

#247

#248

#249

#25

#250

#251

#252

#256

#257

#258

#259

#260

#262

#264

#265

#266

#267

#268

#268

#269

#27

#270

#270

#272

#273

#274

#275

#276

#277

#278

#279

#280

#281

#282

#283

#284

#285

#285

#286

#287

#288

#289

#289

#290

#291

#293

#294

#295

#296

#297

#298

#299

#3

#300

#301

#302

#302

#303

#303

#304

#305

#306

#307

#307

#309

#310

#311

#313

#314

#315

#316

#317

#318

#319

#319

#320

#322

#323

#324

#325

#326

#327

#328

#329

#33

#330

#332

#333

#333

#334

#335

#336

#337

#338

#338

#339

#34

#340

#341

#342

#343

#345

#346

#347

#347

#348

#349

#351

#352

#353

#355

#357

#361

#361

#363

#363

#364

#365

#367

#37

#370

#371

#374

#375

#376

#377

#378

#38

#380

#381

#382

#383

#384

#385

#386

#388

#389

#390

#392

#394

#395

#397

#398

#399

#4

#401

#402

#403

#404

#404

#405

#408

#409

#410

#411

#412

#413

#414

#415

#416

#417

#418

#419

#419

#42

#420

#421

#422

#423

#424

#425

#426

#426

#427

#427

#429

#431

#432

#432

#433

#434

#435

#436

#437

#438

#438

#439

#44

#440

#441

#441

#442

#444

#445

#446

#447

#448

#45

#450

#451

#452

#452

#453

#455

#456

#457

#458

#459

#459

#46

#460

#460

#461

#462

#463

#464

#465

#466

#467

#468

#469

#47

#470

#471

#472

#473

#474

#475

#476

#477

#477

#479

#48

#480

#483

#486

#487

#488

#489

#489

#49

#490

#491

#493

#494

#495

#496

#498

#5

#50

#50

#500

#501

#501

#505

#505

#506

#506

#507

#508

#509

#51

#510

#511

#512

#513

#513

#514

#515

#516

#517

#518

#519

#520

#520

#521

#521

#522

#523

#524

#525

#526

#528

#529

#53

#530

#531

#532

#533

#535

#536

#537

#538

#539

#540

#541

#542

#543

#544

#544

#545

#545

#546

#547

#548

#549

#55

#55

#550

#551

#552

#553

#554

#554

#556

#558

#559

#56

#560

#561

#564

#565

#567

#568

#570

#571

#573

#574

#575

#575

#576

#577

#577

#578

#579

#58

#581

#581

#582

#583

#584

#584

#585

#586

#589

#59

#590

#591

#591

#592

#593

#593

#594

#595

#596

#597

#598

#599

#60

#60

#600

#601

#602

#603

#604

#605

#606

#607

#607

#608

#608

#609

#609

#61

#610

#610

#611

#613

#614

#615

#616

#616

#617

#618

#618

#619

#62

#620

#621

#621

#623

#624

#625

#626

#626

#627

#628

#628

#629

#630

#630

#631

#632

#633

#634

#634

#635

#636

#637

#638

#638

#639

#64

#640

#641

#643

#643

#644

#645

#645

#646

#647

#648

#649

#650

#650

#651

#651

#652

#653

#654

#655

#655

#657

#658

#658

#659

#66

#661

#662

#663

#665

#666

#667

#668

#669

#67

#67

#671

#673

#673

#674

#675

#675

#676

#676

#678

#678

#679

#679

#68

#680

#681

#682

#684

#685

#686

#688

#689

#691

#691

#693

#694

#696

#696

#698

#699

#699

#700

#702

#703

#704

#704

#705

#706

#707

#707

#708

#708

#709

#71

#711

#711

#712

#713

#714

#714

#715

#716

#717

#718

#718

#719

#72

#721

#721

#724

#725

#726

#728

#730

#731

#732

#733

#734

#734

#735

#736

#737

#737

#738

#740

#741

#741

#742

#742

#743

#743

#744

#745

#746

#746

#748

#748

#75

#750

#753

#753

#754

#754

#755

#755

#756

#756

#757

#758

#758

#76

#76

#760

#762

#762

#763

#764

#765

#765

#766

#769

#771

#771

#772

#773

#773

#775

#775

#78

#781

#781

#782

#782

#783

#783

#784

#784

#789

#789

#79

#791

#791

#793

#793

#797

#797

#80

#800

#801

#802

#803

#808

#808

#809

#810

#811

#812

#812

#814

#814

#815

#819

#819

#82

#820

#820

#821

#822

#822

#825

#825

#827

#827

#828

#828

#830

#830

#831

#831

#832

#834

#834

#835

#840

#840

#841

#841

#843

#847

#847

#850

#851

#851

#852

#853

#854

#854

#856

#856

#858

#858

#87

#87

#89

#89

#9

#90

#95

#97

#98

#99

#99

Mono Color

• 1bdedaff52
  Merge 554c5b45cc into f1e2ace651 Evan Owen 2026-07-06 21:54:05 -04:00
• 554c5b45cc
  Add a Mojo GPU port (CUDA + Metal, Mojo 1.0) to notable forks ulmentflam 2026-07-06 21:51:52 -04:00
• bc88c50bf0
  Merge de1dd62018 into f1e2ace651 Jeff Daily 2026-06-12 18:51:16 +00:00
• de1dd62018
  [ROCm] Fix HIP build on systems with partial GCC installs and off-PATH amdgpu-arch (#1) mycpuorg 2026-06-12 11:51:11 -07:00
• 3a3cdaaa3f
  Merge cd609e515f into f1e2ace651 Murad 2026-06-12 05:12:49 +00:00
• cd609e515f fix: the undefined reference compile error when using cuDNN Murad1997 2026-06-12 05:09:19 +00:00
• e33cc58625 [ROCm] Add an AMD GPU (ROCm/HIP) quick start to the README Jeff Daily 2026-06-08 18:50:21 +00:00
• d19a3220d7 [ROCm] Add a HIP/ROCm build for AMD GPUs Jeff Daily 2026-06-04 06:37:19 +00:00
• fa280da9c2 fix: use padded vocab size for wte gradient check remysys 2026-05-22 00:58:40 +08:00
• eed5c8725a
  Merge 8a205a24a2 into f1e2ace651 RoboBryce 2026-05-19 14:14:18 -04:00
• 8a205a24a2 fix two reward-hacking gaps in the bench harness Bryce Adelstein Lelbach 2026-05-19 18:03:00 +00:00
• 654b4ceae5 fix: update cudaMemAdvise host location Caaren Amirian 2026-05-08 04:28:34 -07:00
• 21ae5f4a19
  Merge ea74abf0b3 into f1e2ace651 ziqiaozhou 2026-04-22 13:11:38 -07:00
• ea74abf0b3 Fix possible data race between two writes on a shared memory in softmax_autoregressive_backward_kernel Ziqiao Zhou 2026-04-22 12:46:46 -07:00
• 40d1f7c26e Support for quantized models Cruthaifios 2026-03-22 08:41:31 -05:00
• 17562bf011 Got serving ggufs working Cruthaifios 2026-03-21 22:33:41 -05:00
• 0ca13dbbce
  Merge 93f6cc9754 into f1e2ace651 Rahul Patnaik 2026-03-14 17:57:21 +05:30
• 93f6cc9754 add softmax kernel 9: online softmax with 4x unrolled loads RahulPatnaik 2026-03-14 17:54:03 +05:30
• f8a1e32500
  Merge fd9fb82f37 into f1e2ace651 BigJai 2026-03-03 22:23:00 +11:00
• fd9fb82f37 feat(cpu): add optional BLAS acceleration for matmul forward/backward ENI 2026-03-03 12:22:11 +01:00
• b0f7655d86
  Add init-from-scratch support to train_gpt2.c Matt Langston 2026-03-01 22:10:03 -08:00
• fe539ca9d7 merged all relevant files into 2 Anand Trehan 2025-12-12 02:15:17 -05:00
• 35bc7bfaaa merged microbatch stuff into new file Anand Trehan 2025-12-12 01:49:01 -05:00
• 3218cf0e7e merged microbatch stuff into new file Anand Trehan 2025-12-12 01:21:16 -05:00
• 877e16940d Merge remote-tracking branch 'origin' into microbatching jiuxhrixi Anand Trehan 2025-12-12 00:29:07 -05:00
• 1943eb8c96 removed nccl and used only nvshmem Anand Trehan 2025-12-12 00:23:12 -05:00
• 82a3a6064f Micro batching works! wohoo! mpragnay 2025-12-12 00:10:45 -05:00
• c1b78a6969
  Merge 7c842d8267 into f1e2ace651 Goodnight 2025-12-11 18:16:09 +00:00
• b9fa386528 merged 2 things Anand Trehan 2025-12-11 00:52:46 -05:00
• be13afd49f more than 2 gpu support added ayushyadav99 2025-12-11 00:20:38 -05:00
• bd9ae9e7a5 more than 2 gpu support ayushyadav99 2025-12-11 00:17:44 -05:00
• 9edbfbf3d2
  Remove debug print statement from gpt2_build_from_checkpoint anand-trehan 2025-12-10 23:55:11 -05:00
• b359799d83 partitioned model Anand Trehan 2025-12-10 23:50:58 -05:00
• 2f261ad4d7 print loss on only one mpragnay 2025-12-10 19:43:00 -05:00
• 711082a746 working state Venkata Lakshmi Narasimha Raju Kanumuri 2025-12-10 01:38:15 -05:00
• 5e8ab0285e added print func Venkata Lakshmi Narasimha Raju Kanumuri 2025-12-10 00:32:22 -05:00
• 591569dec8 added ncclSum over ncclAvg Ayush Yadav 2025-12-09 23:00:11 -05:00
• 3833b824ac minor cleanups mpragnay 2025-12-09 22:35:41 -05:00
• 481f23a512 linking with nccl Raju Kanumuri 2025-12-07 00:44:53 -05:00
• 2cc1521185 minor Raju Kanumuri 2025-12-07 00:38:07 -05:00
• 8e0c9b1f14 added some more nccl and all reduce func Raju Kanumuri 2025-12-07 00:36:55 -05:00
• 49ac18a29a add reduce added Raju Kanumuri 2025-12-07 00:31:07 -05:00
• 8b01d5514b Added all reduce to find wte grad Raju Kanumuri 2025-12-07 00:25:57 -05:00
• 70e2e39310
  Merge pull request #1 from mpragnay/raju kvlnraju99 2025-12-06 22:37:30 -05:00
• 3edab5b70e fix 4 Raju Kanumuri 2025-12-06 22:31:47 -05:00
• 4c95738044 fix 3 Raju Kanumuri 2025-12-06 22:28:44 -05:00
• d9790b95d1 fix 2 Raju Kanumuri 2025-12-06 22:22:22 -05:00
• f972a828ab fix code 1 Raju Kanumuri 2025-12-06 22:18:00 -05:00
• d1b9dac7aa Fix syntax error - remove duplicate backward pass code Raju Kanumuri 2025-12-06 22:12:23 -05:00
• c9b01e434b Fix NVSHMEM communication - use putmem/getmem instead of float_put Raju Kanumuri 2025-12-06 22:08:24 -05:00
• a786d876db build changes mpragnay 2025-12-06 21:55:48 -05:00
• de1f17078e Build changes mpragnay 2025-12-06 21:45:17 -05:00
• 07049a62eb NVSHMEM changes Raju Kanumuri 2025-12-06 21:15:15 -05:00
• 98263c1dc8 working setup and testing code for nccl and nvshmem mpragnay 2025-12-06 19:50:39 -05:00
• 395c3d2a47 Init Commit, we work on this file mpragnay 2025-12-06 16:48:07 -05:00
• 1e17d1e9da
  Merge 61a7f6bc1a into f1e2ace651 Mainak Sarkar 2025-10-04 20:00:41 +00:00
• 61a7f6bc1a rename layernorm folder Mainak-learner 2025-10-04 14:55:08 -05:00
• a2b7d64634 correct initialization of activation sizes Mainak-learner 2025-10-04 10:55:23 -05:00
• 6a4ae37130
  Merge 26e65ab9ac into f1e2ace651 Davran Damkhan 2025-10-03 12:52:06 -07:00
• 26e65ab9ac
  Update train_gpt2.cu Davran Damkhan 2025-10-03 14:50:33 -05:00
• 4732ff63a2
  Update train_gpt2.cu Davran Damkhan 2025-10-03 13:23:32 -05:00
• 566d86abee
  Update train_gpt2.cu Davran Damkhan 2025-10-03 12:59:05 -05:00
• f14332b45e fix build error Mainak-learner 2025-10-01 16:24:35 -05:00
• 2b04497a8e fix build error Mainak-learner 2025-10-01 16:19:46 -05:00
• d4dfacf94f fix build error Mainak-learner 2025-10-01 16:12:59 -05:00
• 849208353b fix build error Mainak-learner 2025-10-01 16:05:00 -05:00
• 2a43c5f2fd fix build error Mainak-learner 2025-10-01 16:02:09 -05:00
• 37194b274d fix build error Mainak-learner 2025-10-01 15:46:29 -05:00
• cd9366e9aa fix build error Mainak-learner 2025-10-01 15:40:55 -05:00
• 6839568e36 fix build error Mainak-learner 2025-10-01 15:23:13 -05:00
• 715be94b1f added RoPE/GQA/SwiGLU Davran Damkhan 2025-10-01 14:28:52 -05:00
• a9f2625367 add changes to switch layernorm for rmsnorm Mainak-learner 2025-10-01 12:48:29 -05:00
• 0a34d98813
  Merge e969709118 into f1e2ace651 Ferraronp 2025-09-12 09:52:08 +03:00
• e969709118 Fix offset handling ferraronp 2025-09-12 09:18:13 +03:00
• f64d7fca0f
  Merge 6b41d21649 into f1e2ace651 Philippe 2025-09-11 09:57:58 +02:00
• 6b41d21649 Add Rust port with safety + CUDA to notable forks philippedavid 2025-09-11 09:19:39 +02:00
• 736ce53b1a
  Merge 820ab2eff3 into f1e2ace651 Aroun Demeure 2025-08-24 11:30:17 +01:00
• 820ab2eff3 Fix for WSL by disabling Multi-GPU when only 1 GPU is present (also more efficient in general) Aroun Demeure 2025-08-24 10:41:45 +01:00
• 71ee33f62c Fix for CUDA 13 Aroun Demeure 2025-08-24 10:17:07 +01:00
• 7ed1d93293
  Merge 9f05d7b939 into da88cb11b0 ngc92 2025-07-24 01:23:42 +00:00
• 9f05d7b939 use unsloth version of llama which does not require HF login Erik Schultheis 2025-05-02 22:25:44 +02:00
• b38d73e908
  Merge f51ab0383d into f1e2ace651 Igor Shilov 2025-07-03 19:12:19 +01:00
• f51ab0383d fix np.int32 overflow in token counting Igor Shilov 2025-07-03 17:56:28 +00:00
• 7c842d8267 docs: fix small typos in markdown files goodnight 2025-07-01 12:00:00 +01:00
• b85a2a42ca
  Merge 526e687f85 into da88cb11b0 Insop 2025-06-26 18:03:37 -07:00
• 62e74448c8 ci updates Erik Schultheis 2025-05-01 14:49:53 +02:00
• 1e396aab22 ensure GPT2 compatibility Erik Schultheis 2025-05-01 14:35:08 +02:00
• 94df81d70d GQA cuDNN for llama3 Erik Schultheis 2025-04-08 14:01:56 +02:00
• 9b578eb53e GQA rope Erik Schultheis 2025-04-08 13:50:42 +02:00
• 4052f29c86
  Merge da88cb11b0 into f1e2ace651 Andrej 2025-06-26 17:03:43 +00:00
• da88cb11b0
  Merge pull request #803 from ngc92/ngc92/llama3-tied-weights llama3 Andrej 2025-06-26 10:03:39 -07:00
• 5c17e4e256
  Merge pull request #821 from ngc92/out-of-bounds-bugfix Andrej 2025-06-26 10:01:10 -07:00
• 9caeceb7ae command-line overwrite to forcibly untie embeddings for llama3.2 models Erik Schultheis 2025-04-14 13:22:39 +02:00
• 9688eef519 enable tied embeddings Erik Schultheis 2025-04-14 13:07:37 +02:00
• ffcfe99955 fix out-of-bounds access in rmsnorm kernel Erik Schultheis 2025-06-26 17:52:20 +02:00
• 9c60616276 fix out-of-bounds access in rmsnorm kernel Erik Schultheis 2025-06-26 17:52:20 +02:00
• 5c7ff1acf2
  Merge 79fbfac059 into f1e2ace651 Hai, Nguyen Hoang 2025-06-24 15:08:14 +07:00
• 79fbfac059 feat: add my name for YOLO badge Nguyen Hoang Hai (TGKC-IT-HCM) 2025-06-24 15:07:41 +07:00
• cb3b3c909e
  Merge 198d9148a1 into f1e2ace651 Qchains 2025-06-21 04:26:48 +00:00
• 198d9148a1
  Create PMLL.c Qchains 2025-06-21 00:25:58 -04:00