Why random shuffled sample access, and when it is the wrong target
The question this document answers: CA3 built a layout (§6.5) whose property is that sample i is an address rather than a walk. Is that worth having?
The honest answer has three parts, and only the third is comfortable.
- Training does need shuffling, and the reason is not a preference — it is what stochastic gradient descent assumes.
- Production training at scale does not implement shuffling as random access, and every large system deliberately abandoned it. The mechanism is physical and it is not going away.
- For sequentially-recorded scientific data, the substitutes those systems use demonstrably fail — and that is the case CA3 holds. This is where the layout earns its place, but not in the form we first measured.
Sources are in references.md; quotations are verbatim from
the pages named there.
1. Why shuffling at all
The canonical loop is stated plainly by the AIStore paper (Aizman, Maltby & Breuel, IEEE BigData 2019):
"(1) randomly shuffle the dataset; (2) sequentially iterate through the shuffled dataset."
SGD's convergence argument assumes each minibatch is an unbiased sample of the distribution. When on-disk order correlates with a label — all of one condition written before all of another — consecutive minibatches are drawn from different distributions, and the gradient signal alternates rather than converging. The failure is not slow training; it is a model that learns the recording order.
The consequence is visible, not merely theoretical. scDataset (ICML 2026) reports "periodic loss spikes at plate boundaries" under sequential reading, the signature of catastrophic forgetting as the loader crosses from one experimental batch into the next.
In PyTorch this is shuffle=True on a map-style dataset: a RandomSampler
emits a permutation and __getitem__ is called once per index. That is
per-sample random access, and it is what the textbook describes.
2. Why nobody does it that way at scale
Because a random read below a filesystem's block size does not go faster for being smaller. FFCV (CVPR 2023) states the threshold:
"each file system has a block size after which random reads become sequential reads. In almost all system, this lies below 2MB."
AIStore gives the same fact numerically:
"4KB random read throughput for SSDs ranges anywhere between 200MB/s and 900MB/s, thus at the low end matching sequential read performance of a stock enterprise HDD … performance at scale requires optimized random reading at sizes orders of magnitude greater than 4KB."
This is the single most important thing to understand about §6.5's measurement. A layout that fetches 8.5 MiB where another fetches 8.6 GiB has a real advantage only if the reads are large enough for bytes to be the cost. At 896 bytes a block, they are not — a 896-byte random read costs about what a 4 KiB one costs, and our own mmap measurement showed exactly that: 37.6 MiB of page faults to deliver 8.5 MiB of blocks.
So the systems built for scale replaced random access with sequential reads plus a shuffle approximation:
| System | What it does instead | Its own words |
|---|---|---|
| WebDataset | Sequential tar shards, shuffle buffer, shard-order permutation | "purely sequential I/O pipelines … 3x-10x for local drives compared to random access"; "tar files do not support random access" |
| MosaicML Streaming | Five shuffle algorithms, default py1e, shuffling within blocks | Global naive shuffle is "the least download-efficient of all shuffle algorithms. Training throughput is often much lower" |
| Megatron-Energon | WebDataset underneath, explicit shuffle buffer required | Tuned for "balanced shuffling randomness vs seeking performance impact" |
| FFCV | Quasi-random: permute 8 MB pages, buffer them, draw batches from the buffer | Differs from WebDataset only in granularity — "pages in FFCV are much smaller than WebDataset shards, leading to significantly better randomness" |
And for web-scale corpora that substitution is nearly free. Nguyen et al. (IPDPS 2022), at up to 4,096 workers:
"To much of our surprise, in the overwhelming majority of our experiments local shuffling performs almost identical to global shuffling in terms of validation accuracy."
while "global shuffling on 128 workers is almost 5x slower than local shuffling." CorgiPile (SIGMOD 2022) reaches the same place from theory: block-then-tuple shuffling puts "the gap between Shuffle Once and CorgiPile … below 1 % for the final training/testing accuracy."
If CA3's data looked like ImageNet, this section would end the argument and §6.5 would be solving a problem the field has already routed around.
3. Why it does not end the argument for CA3
CA3 records sessions. On-disk order encodes time, condition, participant and apparatus by construction — §7.2's ordering invariant exists precisely because the file is written in arrival order. That is the property under which shuffle buffers stop working.
scDataset (ICML 2026) is the closest published analogue: scientific recording data, on-disk in HDF5, 100M cells across experimental plates. Four classification tasks, two architectures, three seeds:
"Streaming and streaming with a shuffle buffer achieve similarly poor performance, confirming that a buffer of 16,384 cells does not mitigate bias when plate-scale heterogeneity spans tens of millions of cells."
Drug and mechanism-of-action classification collapsed to near chance under streaming and under a 16,384-sample shuffle buffer. Minibatch label entropy recovered to ~3.61 against ~3.62 for true random sampling only once block sampling was used.
Their own generalisation names CA3's case without knowing it exists:
"time-series sensor data from IoT deployments"
and anything with "natural clustering (spatial, temporal, or organizational)." A recording of one participant's session, written in time order, is that in the strongest form.
And the alternative is not cheap. scDataset measured true random sampling from on-disk AnnData at ~20 samples/sec — over 58 days for one epoch. So the choice on scientific recordings is not "random access or a shuffle buffer, both fine." It is between a shuffle buffer that biases the model and a random access that does not finish.
4. What the field converged on, and what CA3 should target
Block or quasi-random sampling with batched fetching. Four systems reached it independently from different directions:
| The shape | |
|---|---|
| FFCV | Permuted 8 MB pages, buffered |
| scDataset | Block size 16, fetch factor 256 — matched true random on all four tasks |
| CorgiPile | Two-level block-then-tuple, ~2 % of dataset as buffer |
| Nguyen et al. | Local shuffling plus 0.3 partial exchange; each worker holds 0.03 % |
scDataset's decomposition is the useful one: 15× came from batched fetching alone, even on pure sequential streaming (345 → 5,263 samples/sec), before any randomisation was added. The win is in coalescing, not in addressing.
What that means for §6.5
The layout is right and the benchmark's access pattern is wrong.
§6.5's actual property is that a contiguous run of samples is one read at a
known offset, with no parsing. That is exactly what block sampling needs: draw
a random block index, fetch block_bytes × b in one read, shuffle within a
buffer of such blocks. The batched variant in measurement/baselines is that
pattern in embryo and it already fetches a third of MCAP's bytes.
Ten thousand individually shuffled indices is not the workload to optimise for, and measuring it flattered the layout in a way that will not survive review. The benchmark should be restructured to sweep block size and fetch factor, reporting samples/sec, against a take rate swept across orders of magnitude rather than fixed at 10 %.
What it means for MessageOffsets (§7.5)
The same argument applies and lands differently. 0x49 makes a single
message addressable inside a §6.2 chunk. If the target is block sampling, a
consumer wanting a run of adjacent messages needs the offsets of the run's
first and last — which the table gives — and reads the span between them in
one fetch. The record is more useful for coalesced runs than for scattered
singles, and §7.5 should say so.
Is data loading even the bottleneck?
Worth checking before optimising it, and the answer is a qualified yes.
Mohan et al. (PVLDB 2021), nine models on production hardware:
"DNNs spend 10–70 % of their epoch time on blocking I/O, despite pipelining and prefetching, simply because the compute rate is higher than fetch rate"
and up to 75 % for ResNet50 on HDD with distributed training. Google's fleet analysis (tf.data, VLDB 2021) found "20 % of jobs spend more than a third of their compute time ingesting data" and "30 % of the total compute time is spent ingesting data."
The qualification matters. Mohan et al. separate fetch stalls from prep stalls, and prep — decode, augment — frequently dominates: even fully cached, 8-GPU ResNet18 spent "65 % and 50 % of the epoch time on prep stall" in TensorFlow and MXNet. And the bandwidth a GPU actually demands is modest: AIStore measured "~60-70MB/s per GPU" falling to "40-50MB/s per GPU beyond a total of 8 GPUs".
So for CA3 the case is strongest where decode is cheap — which a §5.8 block form is, being a memcpy into a tensor — and where the working set exceeds page cache, which a corpus of session recordings does and a single file does not. Those are the conditions under which a bytes-fetched advantage converts into wall clock. Both should be stated whenever the layout's benefit is claimed.
One anti-sequential finding, recorded because it cuts our way
Mohan et al. again, on why sequential shard formats are not free either:
"TFRecord format results in 40 % higher cache misses than the ideal because, the sequential access nature of TFRecords (and RecordIO) is at odds with LRU cache replacement policy."
An addressable layout can be cached by the sample; a shard format is cached by the shard, and the eviction policy fights it. That is a point in §6.5's favour that our own benchmark did not measure and could.