Ross Wheeler
|
e5083be900
|
Moved windows glob() over to dev/unistd.h
Added header guard and changed long->int64_t in dataloader.h
|
2024-05-24 16:56:52 -07:00 |
|
Ross Wheeler
|
d3f26951ef
|
Add glob() for windows
Tested with tinyshakespeare and fineweb
|
2024-05-23 00:06:18 -07:00 |
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
Andrej Karpathy
|
099d30f814
|
add a super small crappy glob for windows that only matches a single unique file. this will make CI happy but we can't train on sharded data on windows until this is improved
|
2024-05-22 13:08:18 +00:00 |
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
Andrej Karpathy
|
ccc240caab
|
make comment more helpful
|
2024-05-21 18:01:35 +00:00 |
|
Andrej Karpathy
|
1defbd4a19
|
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
|
2024-05-21 17:05:27 +00:00 |
|
Andrej Karpathy
|
f7cbb30316
|
step 2 of dataloader refactor: separate out the dataloader to its own file, change its signature a little bit, and (notably) change from cudaMallocHost to a simple malloc, so that this file doesn't have to be cuda aware
|
2024-05-21 15:32:46 +00:00 |
|