Measurement

Everything this repository measures, and everything it knows about how to measure it. The harnesses are the subdirectories; the documents beside them are what the field already established and what our own runs found. They are in one directory because a number and the case for trusting it are the same claim, and separating them is how a benchmark gets quoted without its caveats.

baselines/	§6.5's columnar layout against §6.2, Parquet and MCAP. A Rust workspace, outside the crate workspace. Decision 0008 carries its numbers, and the review below carries what is wrong with them
read-bench/	Timing and allocation counting on each read path, against any .ca3 file. One binary per question; each question is written up in a note under docs/notes/
dlio/	A BaseDataLoader plugin and workload configurations for MLCommons' DLIO, which is the route to a number neither party has to defend

And the documents:

references.md	Every paper, specification and benchmark suite, with what each one actually says
access-patterns.md	Whether shuffled random access is the pattern worth optimising, and the evidence on both sides
methodology.md	The standard pitfalls, and what credible format comparisons do about them
motivations.md	What CA3 decides, and the published evidence that each problem is real
dlio-results.md	§6.5 against HDF5 and against no container at all, in the two regimes that give opposite answers, plus five defects — four in DLIO and one of ours

Everything here was gathered on 2026-09-09 by reading the sources named. Where a figure came through a summary rather than from the page itself, it is marked as such — those are the ones to check before quoting.

The state of our own measurement, stated first

Since this was written, the DLIO runs happened. dlio-results.md has them, and the answer has two halves. Where the read path is the constraint (8 MiB records, 12 runs per format), §6.5 reads 3.6 % faster than HDF5 at t = 4.8, and is statistically indistinguishable from a container that does not exist — which bounds how much of that can be our own plugin. Where the device is the constraint (144 MiB records, 99 GiB against 59.4 GiB of RAM), all three land within 2.1 % and the container stops mattering at all.

That answers the "self-designed" objection below for the read axis and for HDF5 only. The Parquet and MCAP comparisons in measurement/baselines still carry every confound listed here, and the write axis is still unmeasured.

measurement/baselines compares §6.5's columnar layout against §6.2, Parquet and MCAP. Decision 0008 carries its numbers. A review of the methodology against the field's conventions found four confounds, and the first is disqualifying as the benchmark currently stands.

	The objection
1. The take rate is a scan	10,000 reads of 100,000 samples is 10 % selectivity. At 10 %, a chunked format touches essentially every chunk, so the measurement is per-chunk read amplification rather than random-access capability. Lance's VLDB 2025 evaluation uses 256 random indices against datasets of one billion rows for exactly this reason
2. Bytes fetched is not a cost function	Below a filesystem's block and readahead threshold — "in almost all system, this lies below 2MB" (FFCV) — fetching fewer bytes costs the same as fetching more. A 1,000× advantage in bytes that yields 1.2× in wall clock is a common and real outcome
3. MCAP is not a competitor on this axis	Its own evaluation document says "A recording format should be optimized for write throughput first and foremost", and its published benchmarks measure write throughput with no random-access measurement at all. Comparing a read-optimised layout against an append-optimised container on shuffled reads is the apples-and-oranges pitfall by definition
4. The Parquet baseline was not tuned	Parquet's random-access throughput moves by roughly 64× on row-group and page configuration alone — ~5,500 rows/sec at defaults against ~350,000 with 8 KiB pages. Our baseline used defaults, so that alone could account for most of the gap

What that does and does not overturn. §15's trigger for 0x0E asked whether §6.2's row-wise encoding costs materially more, and the row-versus- columnar half of the comparison is internally consistent and holds — both sides are ours, tuned identically, on the same bytes. What does not survive is the cross-format claim, and it should not be quoted until the changes below are made.

The two changes that would fix it

1. Sweep the take rate, and scale the dataset. Measure at 0.01 %, 0.1 %, 1 % and 10 % on a dataset ten to a hundred times larger, reporting samples/sec through a real torch.utils.data.DataLoader and an I/O operation count rather than bytes alone. If the advantage survives at 10 % on a large dataset, that is a strong result; if it collapses, that is the honest number and reporting it is what makes everything else believable.
2. Run it under DLIO with a custom BaseDataLoader plugin. That converts a self-designed benchmark into a self-run instance of a benchmark MLCommons designed and audits. access-patterns.md and references.md carry the plugin API and the workload shapes.

Then either drop MCAP or reframe it explicitly as the incumbent recording format, which was not designed for this access pattern — which is a fair comparison, stated fairly.

What two adversarial reviews then found

The harness was rebuilt on two axes and handed to two reviewers briefed to break it rather than be fair to it. Both succeeded, and what they found is recorded here rather than repaired quietly, because the pattern matters more than any single fault.

Every bias they found ran the same way — toward CA3. That is what a self-designed benchmark does, and it is why more careful self-design is not the fix.

Methodology

bytes read was measured for CA3 and derived from metadata for Parquet	The one target where the number came from a comment rather than a counter was the baseline
The "tuned Parquet" file contained none of the tuning	set_data_page_size_limit(8192) produced zero 8 KiB pages, verified against the artifact. The tuned row differed only in row-group size
Parquet was denied with_page_index, with_row_selection and with_projection	All three ship in the version depended on. Meanwhile CA3 got a coalescing loop that exists nowhere in crates/container
The page-granular measurement was deleted	memmap2 remains a dependency with a comment explaining page faults; nothing uses it. The 4.4× amplification it once showed survives only in prose
Parquet was charged for a stamp_ns column nobody reads	CA3 skips its stamp region entirely
read_ops was coalesced for CA3 and not for Parquet	1,563 operations reported where the physical answer is 1

Under symmetric accounting the reviewer's estimate is 1.4×–2.7×, not 10×–65×.

Correctness

The checksum that was supposed to police all of this does not work, and this was demonstrated rather than argued. Swapping samples 23001 and 71089 — 811 of 896 bytes different — yields a bit-identical checksum, because only the first four bytes of each block are hashed and the corpus has 283 such collision pairs.

Three defects were also found in the format code, not the harness, and are fixed: a shift that let a column index past the mask alias one inside it; hyperslab returning the whole block for a reversed range, its guard unreachable; and neither region-length invariant enforced despite the documentation requiring both.

The verdict

Write axis	Defensible, with two disclosures
Block-sampling read rows	Defensible after repair, expected to survive at 1.4×–1.8×
Scattered rows and the cross-format read claim	Not defensible. Withdrawn.
§6.2 against §6.5	Sound — both sides are ours, same code, same bytes, no baseline to handicap. This is what §15's trigger asked for and it is met

And the conclusion that follows

More careful self-benchmarking is not the repair. The route to a number neither party has to defend is methodology.md's: a DLIO custom BaseDataLoader, run against MLCommons' own workloads. DLIO ships the baseline readers, so they cannot be handicapped by whoever writes the plugin — which is the structural fix for every row in the table above.

Until that exists, nothing cross-format in this repository should be quoted.

The finding that matters more than the confounds

The access pattern we optimised for is not the one the field uses, and for this format's data that may be the field's mistake rather than ours.

Production training at scale does not do per-sample uniform random access. WebDataset, MosaicML Streaming, Megatron-Energon and FFCV each abandoned it deliberately for sequential reads with a shuffle buffer, and they give the mechanism: random reads below a filesystem's block size do not go faster for being smaller.

But shuffle buffers fail on sequentially-recorded scientific data, which is what CA3 holds. scDataset (ICML 2026) measured a 16,384-sample shuffle buffer producing near-chance accuracy on tasks where true random sampling worked, because on-disk order encoded experimental condition. Their generalisation names our case: "time-series sensor data from IoT deployments" and anything with "natural clustering (spatial, temporal, or organizational)." A CA3 recording is ordered by time and grouped by session, so it has that property by construction.

The resolution everyone reached independently is block or quasi-random sampling with batched fetching — FFCV's 8 MiB pages, scDataset's block 16 / fetch 256, CorgiPile's 2 % buffer, Nguyen's 0.3 exchange rate. Not uniform per-sample random access, and not naive streaming.

So the batched variant in measurement/baselines is closer to the right target than the plain random one, and the benchmark should be restructured around block sampling with a swept take rate rather than around ten thousand individually shuffled indices. access-patterns.md has the evidence.