Skip to main content

References

Gathered 2026-09-09. Each entry says what the source actually claims, because a citation without its claim is a citation nobody checks.

Marked [summarised] where the figure reached this document through a summary rather than from the page itself. Those are the ones to verify before quoting in anything that ships.

Benchmark suites​

MLPerf Storage — MLCommons​

What it measures: how many emulated accelerators a storage system keeps above a utilisation threshold — 90 % for UNet3D and ResNet50, 70 % for CosmoFlow — plus checkpoint read/write bandwidth for Llama-3 at 8B/70B/405B/1T, a vector-database test, and a KV-cache test. It uses no GPUs: accelerator compute is emulated with sleeps while real data is read through PyTorch or TensorFlow.

It does not benchmark file formats. It generates synthetic files whose size distribution matches a real dataset. Access layers are POSIX and S3.

Divisions: CLOSED forbids code changes, for comparability. OPEN permits documented DLIO and configuration changes — where a custom format would go.

DLIO Benchmark — Argonne​

MLPerf Storage is built on it. Formats: tfrecord, hdf5, npz, csv, jpeg, png. Loaders: tf.data, PyTorch DataLoader, synthetic.

file_shuffle and sample_shuffle both default to off.

The plugin path, which is the actionable route to an independent number: implement BaseDataLoader, place it in the plugins directory, then set

data_loader_classname: dlio_benchmark.plugins.experimental.src.data_loader.pytorch_custom_data_loader.CustomTorchDataLoader
data_loader_sampler: index      # or: iterative

index is map-style random access; iterative is sequential streaming. That one setting is the distinction access-patterns.md is about.

The workload shapes, read from the configs themselves:

WorkloadFormatFilesSamples/fileRecord sizeShuffle
unet3d_h100.yamlnpz1681~146 MB (σ 68 MB)file_shuffle: seed, sample_shuffle: seed
cosmoflow_h100.yamltfrecord524,2881~2.8 MBshuffle_size: 2
resnet50_h100.yamltfrecord1,0241,251~115 KBno shuffle keys set

The one audited industry benchmark models random access at whole-file granularity with multi-megabyte records, or sequential streaming inside shards. Not fine-grained shuffled row access inside a large file.

NVIDIA DALI​

dali/test/python/test_RN50_data_pipeline.py — an input-pipeline-only benchmark, not end to end. Covers file reader, TFRecord, RecordIO, LMDB, Caffe2 LMDB and WebDataset, with --read_shuffle and --random_shuffle. Useful as a harness shape; NVIDIA publishes no authoritative cross-loader results table.

FFCV benchmark suite​

https://docs.ffcv.io/benchmarks.html — ImageNet on AWS p3dn.24xlarge: 31,278 img/s (RAM, 512px JPEG q90), 172,801 img/s (RAM, 256px raw), 16,631 img/s (EBS, 512px JPEG q90), 1,956 img/s (EBS, 512px raw).

Only FFCV's own numbers appear in that table, and its paper concedes the limit: "we compare FFCV to the standard PyTorch dataloader rather than specialized solutions like NVIDIA DALI." That disclosure is what keeps it credible, and is the precedent for how to handle a self-designed comparison.

Nothing benchmarks formats independently​

MLPerf Storage benchmarks storage systems; DLIO benchmarks I/O behaviour; every format comparison in the literature is author-designed. That cuts both ways — it is why our benchmark has no referee, and why nobody else's does either.

Is data loading the bottleneck?​

Mohan, Phanishayee, Raniwala, Chidambaram — Analyzing and Mitigating Data Stalls in DNN Training​

PVLDB 14(5):771–784, 2021 — https://vldb.org/pvldb/vol14/p771-mohan.pdf. Nine models, three tasks, four datasets, Microsoft production servers. Artifact: msr-fiddle/DS-Analyzer.

  • "DNNs spend 10–70 % of their epoch time on blocking I/O, despite pipelining and prefetching, simply because the compute rate is higher than fetch rate" — 35 % of dataset cached, SSD + V100
  • "we show that it accounts for up to 65 % of the training time"
  • HDD + 1080Ti, two-server distributed: ResNet50 "stalled on I/O for 75 % of its epoch time"
  • Fully cached, 8-GPU ResNet18: native TF and MXNet spend "65 % and 50 % of the epoch time on prep stall"; DALI still leaves "upto 50 % prep stalls"
  • Against sequential formats: "TFRecord format results in 40 % higher cache misses than the ideal because, the sequential access nature of TFRecords (and RecordIO) is at odds with LRU cache replacement policy"

They separate fetch stalls from prep stalls deliberately, and prep frequently dominates. If a format's samples need little decode and the working set fits in page cache, a bytes-fetched win may not convert into wall clock at all.

Murray, Šimša, Klimovic, Indyk — tf.data: A Machine Learning Data Processing Framework​

VLDB 2021, https://arxiv.org/abs/2101.12127. Fleet analysis over millions of Google jobs.

  • "20 % of jobs spend more than a third of their compute time ingesting data"
  • "we find that 30 % of the total compute time is spent ingesting data"
  • "13 % of jobs read at least 1 TB of input data"; "over 96 % of total compute resources across jobs are spent in jobs that read over 1 TB"

AIStore — Aizman, Maltby, Breuel, High Performance I/O For Large Scale Deep Learning​

IEEE BigData 2019, https://arxiv.org/abs/2001.01858.

  • The canonical model, stated plainly: "(1) randomly shuffle the dataset; (2) sequentially iterate through the shuffled dataset"
  • "4KB random read throughput for SSDs ranges anywhere between 200MB/s and 900MB/s, thus at the low end matching sequential read performance of a stock enterprise HDD … performance at scale requires optimized random reading at sizes orders of magnitude greater than 4KB"
  • Bandwidth actually required: "PyTorch/ResNet-50 initially demands ~60-70MB/s per GPU, the demand drops to more like 40-50MB/s per GPU beyond a total of 8 GPUs" — one HDD feeds about 2.5–3 GPUs
  • Their stated aim: "to measure the end-to-end performance of the whole (compute + storage) system and to avoid common DL-benchmarking pitfalls"

Rinas — Zhong et al.​

https://arxiv.org/abs/2312.02368. "data loading I/O for shuffling can consume up to 85 % of the total training time for models such as ResNet-152 on the ImageNet dataset (~140 GB)"; 30–50 % throughput degradation for language-model training; 59 % and 89 % improvements from intra-batch unordered fetching.

SPDL — Meta​

https://arxiv.org/abs/2504.20067, submitted 2025-04-23, revised 2026-03-10. 74 % faster ImageNet iteration than PyTorch DataLoader, 38 % less CPU, 50 GB less memory, +33 % on free-threaded Python 3.13t.

Methodology​

Raasveldt, Holanda, Gubner, Mühleisen — Fair Benchmarking Considered Difficult​

DBTest'18, June 2018 — https://ir.cwi.nl/pub/27862/27862.pdf. The pitfall taxonomy methodology.md is organised around:

§3.1 Non-Reproducibility · §3.2 Failure To Optimize · §3.3 Apples vs Oranges · §3.4 Over-specific Tuning · §3.5 Cold/Warm/Hot Runs · §3.6 Ignoring Preprocessing Time · §3.7 Incorrect Code

Lance — VLDB 2025​

https://arxiv.org/abs/2504.15247. [summarised] — verify before quoting.

  • Parquet random access moves ~64× on configuration: ~5,500 rows/sec at defaults against ~350,000 with 8 KiB pages
  • Uses 256 random indices per take, against one billion rows for small types
  • "benefits of coalesced access are easily overstated in benchmarking and less frequent in real-world use cases"

Zeng et al. — columnar format evaluation​

Artifact: XinyuZeng/EvaluationOfColumnarFormats. Found ORC issues "≈4× S3 GET than Parquet", and that request count rather than bytes reverses the SSD ranking on object storage.

MCAP evaluation​

John Hurliman, Foxglove, 2021-10-06 — https://mcap.dev/files/evaluation.pdf. "A recording format should be optimized for write throughput first and foremost." Published benchmarks (https://mcap.dev/guides/benchmarks/rosbag2-storage-plugins) measure write throughput on ramdisk and contain no random-access measurement.

Shuffling: cost and consequence​

Full argument in access-patterns.md; the sources:

SourceWhereWhat it establishes
WebDatasethttps://github.com/webdataset/webdataset"purely sequential I/O … 3x-10x for local drives compared to random access". Shuffling is a buffer plus shard permutation; "tar files do not support random access"
MosaicML Streamingdocs, blog 2023-02-09Five algorithms, default py1e; shuffle_block_size defaults to max(4_000_000 // num_canonical_nodes, 1<<18). Global naive shuffle is "the least download-efficient … Training throughput is often much lower"
Megatron-Energonhttps://nvidia.github.io/Megatron-Energon/Requires explicit shuffle_buffer_size and max_samples_per_sequence "because the library is designed to work on (sequential) webdatasets"; ~100 recommended for both as "balanced shuffling randomness vs seeking performance impact" [summarised]
FFCVLeclerc et al., CVPR 2023, https://arxiv.org/abs/2306.12517"each file system has a block size after which random reads become sequential reads. In almost all system, this lies below 2MB." Uses quasi-random sampling over permuted 8 MB pages
Nguyen et al.IPDPS 2022, https://bgerofi.github.io/papers/tnguyen-IPDPS22.pdfLocal vs global shuffling at up to 4,096 workers. See below
CorgiPileSIGMOD 2022, https://arxiv.org/abs/2206.05830Block-then-tuple shuffle; "the gap between Shuffle Once and CorgiPile is below 1 %"; 1.5× faster than full-shuffle PyTorch on ImageNet
scDatasetICML 2026 (PMLR 306), https://arxiv.org/abs/2506.01883Scientific recording data. See below — the closest analogue to CA3's situation
BatchWeavehttps://arxiv.org/abs/2605.09994 v2, 2026-05-15"a static, pre-processed dataset is shuffled offline, partitioned into fixed-size batches, and streamed to the training process at a predictable rate"
GetBatchAizman, Gaikwad, Żelasko, https://arxiv.org/abs/2602.22434, 2026-02-27Still targets randomised fetch as the workload

A "20–40 % of training time" figure was attributed to BatchWeave by a summarising tool. The PDF was searched and it is not there. Do not use it.

Nguyen et al., IPDPS 2022 — the accuracy cost of local shuffling​

Up to 2,048 GPUs on ABCI, 4,096 nodes on Fugaku.

  • "To much of our surprise, in the overwhelming majority of our experiments local shuffling performs almost identical to global shuffling in terms of validation accuracy"
  • ImageNet-1K/ResNet-50 at 1,024 workers: local matches global "when each worker only processes a very small fraction of the dataset (e.g. 1/1024)"
  • Where it breaks: 9 % gap at 2,048 GPUs on ImageNet-1K; up to 30 % on ImageNet-50 at 128 GPUs (10 % even at 32); 3 % on ImageNet-21K upstream
  • The repair: partial exchange at rate 0.3 restores global-shuffle accuracy at 4,096 workers while each worker stores 0.03 % of the dataset
  • The cost avoided: "global shuffling on 128 workers is almost 5x slower than local shuffling"; DenseNet I/O 19.6 s vs 8 s per epoch at 512 workers

scDataset, ICML 2026 — where shuffle buffers fail​

Single-cell omics in HDF5/AnnData; Tahoe-100M, 100M cells, 314 GB. Code: https://github.com/scDataset/scDataset.

  • True random sampling from on-disk AnnData: ~20 samples/sec — over 58 days for one epoch
  • Block sampling (block b) with batched fetching (fetch factor f): 204× over AnnLoader at b=1024, f=1024; 47× over HuggingFace Datasets; 25× over BioNeMo-SCDL; 15× from fetch-factor batching alone on pure sequential streaming (345 → 5,263 samples/sec)
  • The accuracy result, four tasks × two architectures × three seeds: "Streaming and streaming with a shuffle buffer achieve similarly poor performance, confirming that a buffer of 16,384 cells does not mitigate bias when plate-scale heterogeneity spans tens of millions of cells." Drug and mechanism-of-action classification collapse to near chance under both. Block shuffling at b=16, f=256 matches true random sampling on all four, with minibatch plate-label entropy ~3.61 against ~3.62 for true random
  • Streaming also shows periodic loss spikes at plate boundaries; block shuffling does not
  • Their generalisation names CA3's case: "time-series sensor data from IoT deployments", and anything with "natural clustering (spatial, temporal, or organizational)"

Tooling notes​

torchdata DataPipes are gone. Deprecated in 0.8.0 (July 2024), deleted in 0.10.0. The repository is now "an iterative enhancement of torch.utils.data.DataLoader"; the surviving artifact is StatefulDataLoader for mid-epoch checkpointing. Pin torchdata==0.9.0 if the old API is needed. https://github.com/meta-pytorch/data/releases

shuffle=True has no effect on an IterableDataset — https://docs.pytorch.org/docs/2.14/data.html.

Reference datasets. Vision converged on ImageNet-1K (~140–146 GB), which is why FFCV, DALI, WebDataset, AIStore and CorgiPile numbers can be argued about at all. For scientific data, Tahoe-100M through scDataset's public code is the newest reusable reference, with AnnLoader, HuggingFace Datasets and BioNeMo-SCDL baselines already established.

Could not be verified​

  • Any published cross-loader throughput table putting FFCV, DALI, WebDataset and PyTorch DataLoader on the same axes. docs.ffcv.io shows FFCV only, and the FFCV paper says it did not compare against DALI
  • Any MCAP random-read benchmark — none exists
  • The "20–40 % of training time" figure attributed to BatchWeave — searched and absent