Skip to main content

How to measure this without being wrong

The pitfall taxonomy is Raasveldt, Holanda, Gubner & Mühleisen, Fair Benchmarking Considered Difficult (DBTest'18). Each section below names the pitfall, then what measurement/baselines does about it — including where it still falls in.

§3.2 Failure to optimize the baseline​

The most common way a format comparison lies, and the one our Parquet baseline currently falls into.

Parquet's random-access throughput moves by roughly 64× on row-group and page configuration alone: ~5,500 rows/sec at defaults against ~350,000 with 8 KiB pages (Lance, VLDB 2025 — [summarised], verify before quoting). Our baseline used defaults with a row group of 1,000. That alone could account for most of the measured gap.

The rule: sweep the tuning knobs on both sides, or state plainly that you did not and that the baseline is therefore a floor rather than a fair fight.

FFCV's paper is the precedent for handling this honestly when a full sweep is out of reach — it concedes "we compare FFCV to the standard PyTorch dataloader rather than specialized solutions like NVIDIA DALI" rather than implying it beat them.

§3.3 Apples vs oranges​

MCAP is on our table and should not be there in that form. Its own evaluation document (Foxglove, 2021-10-06) says:

"A recording format should be optimized for write throughput first and foremost."

Its published benchmarks measure write throughput on a ramdisk and contain no random-access measurement at all. Comparing a read-optimised columnar layout against an append-optimised container on shuffled reads is this pitfall by definition.

The repair is not to drop it but to label it: the incumbent recording format in robotics, which was not designed for this access pattern. That is a fair and interesting comparison, and stating the asymmetry is what makes it fair.

§3.5 Cold, warm and hot runs​

Our timings are warm — every target reads bytes already resident. That was a deliberate choice to isolate the layout, and it is stated in the harness.

But cache state changes rankings, not just magnitudes. Mohan et al. found page-cache thrashing causing "85 % of the dataset fetched from storage every epoch" when only 65 % should have been, which reorders formats rather than scaling them.

A file larger than page cache could not be written on the machine that ran this — 63.8 GiB of RAM against 38.5 GiB of free disk — so bytes fetched was counted instead, as a cache-independent proxy. That is a reasonable substitute and it is not a cold measurement; see the next section for why the proxy is weaker than it looks.

Bytes fetched is not a cost function​

Not in Raasveldt's list, and it should be. Below a filesystem's block and readahead threshold — "in almost all system, this lies below 2MB" (FFCV) — fetching fewer bytes costs the same as fetching more. Random 4 KiB SSD reads run at 200–900 MB/s against multi-GB/s sequential (AIStore).

A 1,000× advantage in bytes that yields 1.2× in wall clock is a common and real outcome. On object storage the currency is requests rather than bytes at all: Zeng et al. found ORC issuing "≈4× S3 GET than Parquet", which reverses the SSD ranking.

The rule: report an I/O operation count beside the byte count, and report wall clock through a real loader. Bytes alone is a structural argument, not a performance claim, and should be described as one.

Take rate is an axis, not a constant​

Our benchmark reads 10,000 of 100,000 samples — 10 % selectivity — and calls it random access. At 10 %, any chunked format touches essentially every chunk, so what is measured is per-chunk read amplification rather than random-access capability.

Lance's evaluation avoids this deliberately: 256 random indices against datasets of one billion rows, with the stated reason that "benefits of coalesced access are easily overstated in benchmarking and less frequent in real-world use cases" ([summarised]).

The rule: sweep selectivity across orders of magnitude — 0.01 %, 0.1 %, 1 %, 10 % — and scale the dataset until chunk-granularity effects can appear. A result that holds at 10 % on a large dataset is strong; one measured only at 10 % on a small one cannot distinguish the layout from the chunk size.

§3.1 Non-reproducibility, and §3.7 incorrect code​

Publish the artifact. Every credible comparison in this area does: msr-fiddle/DS-Analyzer (Mohan), XinyuZeng/EvaluationOfColumnarFormats (Zeng), scDataset/scDataset. measurement/baselines is in this repository for that reason.

Check that every implementation returns the same answer. The harness computes an index-weighted checksum per target and prints every target's, not only the failures — a reader that returned the right blocks in the wrong order disagrees rather than passes, and a line that appears only on failure cannot be told from a check that never ran.

Two errors were caught this way before any number was quotable, and both had flattered CA3: the Parquet target opened the file inside the random loop (10,000 open() calls, reported as 6.9 s), and MCAP's writer defaulted to zstd, putting its file below the raw tensor size. Both are recorded in the harness rather than silently fixed.

§3.6 Ignoring preprocessing​

Our scan and random columns measure reaching a sample, not copying it out. A loader that copies the whole block pays the same memcpy in every row.

This matters more than it sounds, because prep frequently dominates fetch. Mohan et al. separate the two and found that even fully cached, 8-GPU ResNet18 spent "65 % and 50 % of the epoch time on prep stall" under TensorFlow and MXNet.

A format whose samples need little decode has a real advantage here — a §5.8 block form is a memcpy into a tensor — and that is worth measuring directly rather than assuming.

End-to-end, or it did not happen​

Every credible comparison ends with a wall-clock training time or an accuracy number, not a micro-benchmark: FFCV, CorgiPile, scDataset, AIStore, Mohan et al. AIStore states the aim explicitly — "to measure the end-to-end performance of the whole (compute + storage) system and to avoid common DL-benchmarking pitfalls."

The route to a number someone else designed​

DLIO accepts a custom data loader as a plugin. Implement BaseDataLoader, place it in the plugins directory, and configure:

data_loader_classname: dlio_benchmark.plugins.experimental.src.data_loader.pytorch_custom_data_loader.CustomTorchDataLoader
data_loader_sampler: index      # map-style random access
# data_loader_sampler: iterative   # sequential streaming

Then run the bundled unet3d_h100, cosmoflow_h100 and resnet50_h100 workloads with CA3 substituted, reporting accelerator utilisation against the thresholds MLPerf Storage publishes — 90 % for UNet3D and ResNet50, 70 % for CosmoFlow.

That converts a self-designed benchmark into a self-run instance of a benchmark MLCommons designed and audits. It is the single highest-leverage change available, and MLPerf Storage's OPEN division exists to accept exactly this kind of submission with full disclosure.

Note what those workloads actually model, because it is not what we measured: whole-file random access at multi-megabyte records (unet3d: 168 files, one sample each, ~146 MB), or sequential streaming inside shards (resnet50: 1,024 files, 1,251 samples each, no shuffle keys set). Fitting CA3 to them is itself informative.