CA3 §6.5 under DLIO

The measurement methodology.md asked for: CA3's derived form against HDF5, under a benchmark this repository did not design.

Two adversarial reviews of the in-house harness found the same defect both times — every bias ran toward CA3, because one author chose the format, the workload, the access pattern, the metric, and both baselines' configurations. The instruction was to stop self-designing and run under DLIO, which MLCommons audits as MLPerf Storage. DLIO picks the workload, the sampler, the batch size, the thread count and the metric; it ships its own HDF5 reader, so the baseline is Argonne's. CA3 supplies only get_sample, through the reader_classname plugin seam in measurement/dlio/ca3_reader.py.

A third reader was added to make the comparison interpretable. measurement/dlio/raw_reader.py reads flat files with no container at all — no header, no framing, no index. Whatever it reaches is what Python file I/O costs when the format costs nothing, and both real formats are then read as a distance from that floor. Without it, "our format is faster" and "we wrote a tighter loop" are the same measurement.

Two regimes, and they answer different questions

The same three readers give opposite-looking answers depending on what the bottleneck is, and the pair is the result rather than either half.

	records	dataset	bounded by
A. reader-path	8 MiB	1.07 GB, page-cache resident	CPU in the read path
B. bandwidth	144 MiB	99 GiB against 59.4 GiB of RAM	the SSD

Regime A — what the container costs in CPU

128 files of 8 MiB, batch 8, 4 read threads, file and sample shuffle on, 20 epochs, 12 independent runs per format.

	samples/s (mean ± sd, n=12)	range	% of the zero-format ceiling
raw, no container	564.06 ± 9.01	547.8 – 576.3	100 %
CA3 §6.5	558.48 ± 10.71	542.5 – 578.2	99.0 %
HDF5, DLIO's own reader	538.99 ± 9.11	525.0 – 558.0	95.6 %

Welch's t-test on the three pairs:

comparison	t	df	verdict
CA3 vs HDF5	4.80	21.4	significant (+3.6 %)
raw vs HDF5	6.78	22.0	significant
raw vs CA3	1.38	21.4	not established

The third row carries the argument. CA3 is statistically indistinguishable from reading files that have no format at all, which bounds how much of its result can be plugin skill: nothing can beat "no container", and CA3 already matches it. HDF5 is significantly below both.

Mechanistically, §6.5 costs one multiplication — sample i begins at blocks_at + block_bytes × i, so the reader seeks once and reads one block with no parse between the file and the array. h5py resolves a dataset path, consults a chunk cache, and copies through the HDF5 library. All three move identical bytes, enforced per-sample by measurement/dlio/verify_identical.py.

Conditionally, this holds where the read path is the constraint: records small enough, or storage fast enough, that CPU decides. Regime B is what happens when that stops being true.

Regime B — what the container costs when the device saturates

704 files of 144 MiB (99 GiB, against 59.4 GiB of RAM), batch 2, 4 read threads, 3 epochs, 3 runs per format. Formats run one at a time and are converted in place, because three copies at this size do not fit on the disk.

	rep1	rep2	rep3	mean ± sd	MB/s
HDF5	15.2 (cold)	19.3	19.3	17.92 ± 2.38	2580
CA3 §6.5	19.1	19.0	19.2	19.12 ± 0.07	2753
raw, no container	19.2	19.5	19.9	19.52 ± 0.32	2811

All three land within 2.1 % of each other, and so does a container that does not exist. At 144 MiB per record, CA3's framing is 98 bytes out of 150,994,944 — 0.00007 % — and the pipeline is saturating the SSD at ~2.8 GB/s. Nothing a format does can register against that.

So CA3's advantage is real and bounded: it lives in regime A. Once records are large enough that the device is the constraint, no container helps, and claiming a win here would be claiming credit for the SSD.

The ordering inside regime B is cache state, not format

Do not read the 2.1 % spread as a ranking. Three things confound it, all recorded rather than corrected:

• HDF5's rep1 (15.2) is the only genuinely cold pass in the table. Every format is measured immediately after being written — HDF5 by the generator, CA3 and raw by the conversion that produced them — so each starts partly resident. HDF5 alone had 35 minutes between its write and its first read.
• Cache warming saturates, and the saturated value is reproducible. HDF5 reps 2 and 3 agree to three significant figures (19.3, 19.3), and CA3 never varies at all (19.1, 19.0, 19.2). The equilibrium is roughly 60 GB of the 99 GiB resident.
• raw was still warming when the sequence ended (19.2 → 19.5 → 19.9), which is most of why it tops the table.

This is not a cold-storage measurement and must not be quoted as one. At 99 GiB against 59.4 GiB the majority of reads reach the SSD, but a large minority do not. The epoch timings show it directly: 60.7 s cold, then 47 s once warm, a 23 % speedup that is page cache and nothing else. A genuinely cold measurement needs a dataset well past 128 GiB, or the ability to drop the cache between epochs — and Windows offers no supported way to do the latter.

What neither regime establishes

Single-sample-per-file only. num_samples_per_file: 1 is forced by a DLIO defect (below), so §6.5's addressing within a large chunk — the property the record was designed for — is exercised by the chunk-table walk but never by a long block region. That is the measurement most worth adding next.

Whole samples only. Nothing here reads part of a sample, so hyperslabs, projections and compressed chunks are all untested.

AU is 1–12 %, and that is the intended condition rather than a failure. computation_time is 1 ms so the simulated accelerator demands data faster than any storage supplies. At unet3d's own 0.323 s every format reaches AU 100 % and the comparison cannot separate them at all; train_au_meet_expectation: fail here means the run is storage-bound, which is the point. measurement/dlio/sweep_au.py turns this into the question MLPerf Storage exists to answer — the fastest accelerator each format holds above 90 % — and has not yet been run at either size.

The defect that mattered most was ours

The HDF5 and CA3 runs were reading different bytes, and an earlier version of this document stated they read the same ones.

ca3_hdf5_stress.yaml carried generate_data: True, the run was launched with ++workflow.generate_data=False to suppress it, and DLIO ignores ++ overrides — so it regenerated every HDF5 file with fresh random content while the CA3 copies still held the previous generation's. Both were 8,386,816 bytes per sample, both were valid files, both benchmarked without complaint. 128 of 128 files mismatched and only a hash could see it.

The timing was very likely unaffected — incompressible random bytes of equal length either way, no compression — but the claim was an unchecked property written down as an established one. Every number in this document was re-measured after the fix; the regime A table is 12 fresh runs per format, not the earlier three.

measurement/dlio/verify_identical.py now hashes every sample in every format and exits non-zero on any mismatch. convert_in_place.py performs the same check before it releases a source file, because --delete-source removes the only thing an output could be compared against. Run the verifier before measuring anything.

Four defects found in DLIO, and what each cost

Each is a claim to re-derive rather than trust; the file and line are given so a reader can check whether a later version still has it.

1. record_length_bytes, not record_length (utils/config.py:903). The key was renamed and only the new spelling is parsed, so a config using the old one silently takes the 64 KiB default. This produced an 18 MB dataset where 1 GiB was requested, and the first runs measured DLIO's startup rather than storage. DLIO's own shipped workload configs still use the old spelling.

2. record_length is then clobbered to 1 (utils/config.py:553).

# hdf5 specific derivations
self.record_length = np.prod(self.record_dims) * self.record_element_bytes

The comment says HDF5 but no format guard exists, and record_dims defaults to []. np.prod([]) is 1.0, so record_length becomes 1 byte for any config that does not set record_dims — after record_length_bytes was read correctly. Because statscounter.py:208 derives I/O throughput as samples/s × record_size rather than measuring it, the reported "Training I/O Throughput (MB/second)" was wrong by a factor of 8.4 million in one run and right in the next. Setting record_dims explicitly fixes it, and the configs here do: 19.2 samples/s × 144 MiB = 2768 MB/s, which is what DLIO now reports.

3. ++ command-line overrides do not reach the config. DLIO reads the YAML dict rather than Hydra's resolved config, so ++train.computation_time= and ++workflow.generate_data= are accepted and ignored — a run reports set value: {'mean': 0.323} regardless. Every variant therefore needs its own file, which is why the *_stress and *_huge configs exist rather than flags. This defect is also what silently regenerated the dataset above.

4. main.py:230 divides a file count by a sample count.

samples_per_step = num_samples_per_file * batch_size * comm_size
aligned          = (num_files // samples_per_step) * samples_per_step

With more than one sample per file this demands more files than any real workload has, trims the file list to zero, and the run reads nothing. DLIO's own resnet50_h100 config trims to zero under it. This is what forces num_samples_per_file: 1.

Two results that reversed, and why they are recorded

A 3-epoch run said the opposite of a 20-epoch run. It reported HDF5 at 912 samples/s against CA3 at 514 — HDF5 1.77× ahead. It was noise, and the standard deviation said so: 376 on a mean of 912, or 41 %. At 128 samples per epoch and roughly 900 samples/s the measurement window is 0.14 s, which on Windows with spawn workers is dominated by process startup rather than reading. Widening to 20 epochs dropped HDF5's standard deviation to 37.5 and reversed the ordering.

A 3-run comparison claimed a separation that 12 runs did not support. On the verified data, three runs per format gave CA3 +3.1 % over HDF5 with overlapping ranges and t = 1.4 — not established. Twelve runs gave +3.6 % at t = 4.8. The conclusion survived; the evidence for it at n=3 did not exist, and the earlier "3 of 3, no overlap" phrasing was an artifact of a small sample.

Both are recorded because each was one report away from being published, and nothing about either looked wrong except a number nobody had to read.

Reproducing

pip install h5py torch
pip install "dlio_benchmark @ git+https://github.com/argonne-lcf/dlio_benchmark@main"

The PyPI 2.0.0 release imports dftracer.logger, which no published pydftracer provides; install from git.

Regime A, from a directory where DLIO has generated its HDF5 dataset:

export PYTHONPATH=measurement/dlio
python measurement/dlio/generate_ca3.py <data-dir>

Then convert the flat control, check the three agree, and run:

python measurement/dlio/generate_raw.py <data-dir>
python measurement/dlio/verify_identical.py <data-dir>

Regime B, which is destructive by necessity and needs ~110 GB free:

python measurement/dlio/run_out_of_cache.py <data-dir> 3

generate_ca3.py and convert_in_place.py copy DLIO's own samples rather than generating any, so every run reads the same bytes. The CA3 files are named .npz and the flat files .npy because FormatType is a closed enum with no ca3 member; the declared format decides the filename extension and reader_classname decides what parses it (utils/config.py:294).

A note for anyone extending this: run DLIO with its output redirected to a file, never through subprocess.run(capture_output=True). The spawn workers inherit that pipe and communicate() waits for an EOF that cannot arrive while a stuck grandchild holds the write end, so a run that died on its first batch presented instead as a process sitting at 0.0 MB/s of I/O and 3.3 seconds of CPU across seventeen minutes, with the exception never reaching anyone.