| .. |
|
adamw.cuh
|
fix bug due to bf16 adamw mv
|
2024-10-01 21:41:17 +00:00 |
|
attention.cuh
|
set stream for attention softmax
|
2025-05-04 22:10:38 +02:00 |
|
cublas_common.h
|
Refactor matmul to only use cuBLASLt and optionally use fused GELU
|
2024-06-30 20:45:37 +01:00 |
|
cuda_common.h
|
BF16 opt state (m/v) with stochastic rounding, seems to work really well (OPTIMIZER_LOW_PRECISION=1)
|
2024-10-01 18:42:38 +00:00 |
|
cuda_utils.cuh
|
BF16 opt state (m/v) with stochastic rounding, seems to work really well (OPTIMIZER_LOW_PRECISION=1)
|
2024-10-01 18:42:38 +00:00 |
|
cudnn_att.cpp
|
Including unistd.h in cudnn_att.cpp for latest MSVC compiler
|
2024-07-20 11:57:10 -07:00 |
|
cudnn_att.h
|
move all kernels into a main_stream in preparation for future parallelism
|
2024-06-04 22:35:12 +03:00 |
|
dataloader.h
|
add support for dataloader to serve uint32_t tokens, as necessary in Llama 3
|
2024-09-16 19:44:33 +00:00 |
|
encoder.cuh
|
take out debugging stuff. we can now run training loop for both models. they don't match yet
|
2024-10-01 17:19:39 +00:00 |
|
fused_classifier.cuh
|
switch losses to fp32
|
2024-06-26 20:48:08 +03:00 |
|
gelu.cuh
|
Fix guarding logic, refactor
|
2024-06-17 14:12:05 +02:00 |
|
global_norm.cuh
|
update norm computation: multi-process reduce on GPU
|
2024-06-25 21:28:52 +03:00 |
|
layernorm.cuh
|
integrate our rmsnorm backward and move the other rmsnorm functions into rmsnorm.cuh that is a new file
|
2024-09-26 22:27:58 +00:00 |
|
logger.h
|
Add learning rate and grad norm logging
|
2024-06-07 17:34:42 +02:00 |
|
matmul.cuh
|
fix default for test_gpt2cu
|
2024-06-30 20:45:37 +01:00 |
|
mfu.h
|
disable if header is not found
|
2024-07-29 10:07:37 +02:00 |
|
outlier_detector.h
|
oops i was only using window size 16 for testing let's make it 128 or something
|
2024-06-25 01:56:04 +00:00 |
|
rand.h
|
Dataloader test and ptx/sass file generator
|
2024-06-21 17:23:03 -07:00 |
|
repkv.cuh
|
added missing stream argument for repkv_backward
|
2025-05-04 20:15:30 +02:00 |
|
rmsnorm.cuh
|
fix out-of-bounds access in rmsnorm kernel
|
2025-06-26 17:52:20 +02:00 |
|
rope.cuh
|
just pushing what i have. it's epsilon away from working sigh. basically at this point of where prints happen, gradients match. but once we backward attention, rope and repkv, gradients don't match. attention hasn't changed so that can't be wrong (?), so it's either repkv or rope. i have to go slower and double check the backward pass of both of these in detail. also had to introduce one more additional buffer for backward
|
2024-09-27 03:37:53 +00:00 |
|
sampler.h
|
refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well
|
2024-06-03 18:10:33 +00:00 |
|
schedulers.h
|
small touchups and add the wsd schedule
|
2024-06-21 00:49:01 +00:00 |
|
swiglu.cuh
|
i can backward through MLP block. Attention block is next
|
2024-09-27 00:21:43 +00:00 |
|
tokenizer.h
|
made implicit includes of standard headers explicit
|
2024-06-04 13:22:16 +03:00 |
|
utils.h
|
Move WIN32 code into unistd.h
|
2024-07-20 12:05:20 -07:00 |
|
zero.cuh
|
fix: fully ignore biases
|
2025-04-14 11:33:34 +02:00 |
