What CA3's design decisions cost elsewhere
spec/container.md argues each of its decisions from first principles, which
is right for a specification: a contract that leans on somebody else's paper
ages badly. This document is the other half — the published evidence that the
problems those decisions address are real, measured, and still unsolved
elsewhere. It is for the reader who asks "why does the format bother", and
for anyone deciding whether a decision is still worth its cost.
Sources and full quotations: references.md. Where a figure
came through a summary rather than the page itself it is marked
[summarised] and should be verified before it is quoted anywhere that
ships.
Nothing here changes what the specification requires. Where this document
and spec/container.md disagree, the specification is right and this is stale.
§5.6 — Absence travels in a validity channel, never as a sentinel
What the spec says. "A tensor slot always holds a number, and (0, 0, 0)
is a legitimate position. Encoding 'not tracked' as zero makes a real
measurement and a missing one the same bytes, and a model trained on that
learns the origin is a place people look."
What everyone else does. NWB's best practice encourages NaN as the
missing-data sentinel, and its only per-sample mechanism is control, one
uint8 per timepoint for a whole series — not per channel. Its
invalid_times is interval-granular and file-level. Arrow has a null bitmap,
which is spec, but null carries no reason code: it cannot distinguish
"sensor dropped out" from "clipped" from "structurally inapplicable". Parquet,
Zarr, HDF5, LeRobot and MCAP have nothing at all. Only CF/netCDF standardises
missingness (_FillValue, valid_range) alongside units and coordinate
systems.
Why it matters more for training than for analysis. An analyst notices an implausible zero. A model does not — it fits it. scDataset's result is the closest published demonstration of what biased sampling does to a scientific recording: drug and mechanism-of-action classification collapsed to near chance when the data reaching the model was systematically unrepresentative, while the same architectures on the same data reached full accuracy once the bias was removed. A sentinel is a smaller version of the same failure, and it is invisible in the loss curve.
Standing cost of the decision. A schema carries a parallel channel, and §15.2 records that nothing yet checks the "implies in both directions" condition a whole-field channel needs.
§5.4 and §5.5 — Units are UCUM codes, and a frame declares its handedness
What the spec says. "Camera space is not a frame. It is a family of frames
that disagree about which way y points, and the disagreement is invisible in
the numbers." §5.5 names the incident: "An integration once negated the y
component of an eyeball centre because the convention was copied from a
rendering package."
Where the field is. Surveyed 2026-09-09:
| Units | Coordinate frame | Per-sample validity | |
|---|---|---|---|
| CF/netCDF 1.13 | required, UDUNITS-2 | grid_mapping | _FillValue, valid_range |
| NWB 2.11.0 | required, free text | absent | interval only |
| Parquet 2.11+ | absent | crs, default OGC:CRS84 | Arrow null bitmap |
| Zarr v3.1 | absent | absent | fill_value (storage, not semantic) |
| HDF5 2.2 | absent | absent | absent |
| BIDS 1.11.1 | convention (SHOULD) | modality sidecars | n/a in TSV |
| ROS 2 / MCAP | convention (REP-103) | frame_id, tree in tf2 | absent |
| LeRobot v3.0 | absent | absent | absent |
Only CF/netCDF standardises all three. NWB's SpatialSeries.reference_frame
is a free-text string with no handedness field, no axis order, no origin and no
transform graph — a prose sentence where a Y-up left-handed engine meets a Z-up
right-handed analysis stack.
Two things worth stealing rather than re-deriving. NWB's unit with
conversion and offset — stored × conversion + offset yields the declared
unit — is a better design than most ad-hoc conventions and lets a producer
write raw ADC integers honestly. And Parquet 2.11.0 (2025-03-21) added a
crs field defaulting to OGC:CRS84 inside its GEOMETRY logical type: a
coordinate-frame declaration accepted into a mainstream columnar format's type
system, shaped as an optional string with a spec-defined default rather than a
required rich object. That is precedent for how such a field gets adopted.
And a caution on vocabularies. A peer-reviewed comparison of eight unit ontologies found only 17 units occur across all of them, and fewer than 75 % are common between any pair. Choosing UCUM rather than inventing is the whole of §5.4's argument, and the alternative is worse than it looks.
§10 — Every stream carries its own clock domain, with uncertainty
What the spec says. G2: "Every timing claim carries its own confidence, computed one way everywhere." G3: "Raw timestamps survive to the reader."
What the training formats do instead, verified in source. LeRobot is the
dominant robotics training format — 75,029 datasets tagged on Hugging Face
[summarised: the count moves and the tag is free-text] — and
dataset_writer.py:225 reads:
frame_index = self.episode_buffer["size"]
timestamp = frame_index / self._meta.fps
There is no wall-clock capture time anywhere in the format. One global
scalar fps; real sensor timing is discarded at add_frame(). Per-feature
fps keys exist in info.json and the reader ignores them. The issue asking
for multi-rate support has been open and unanswered since 2025-05-30.
RLDS has no timestamp field at any level — everything in a step is implicitly synchronous. Archived 2025-11-29.
NWB has one time zero per file. timestamps_reference_time is "time zero
of all timestamps", and the sync group is documented as "lab-specific …
as provided directly from hardware devices" — deliberately unstandardised.
Its model asks you to resolve every clock to one master timeline before
writing, and to discard the evidence of how you did it.
Rerun states the structural limit plainly: Parquet "cannot handle overlapping row groups needed for multi-rate data."
The formats that get it right are capture formats, and they agree with §10.
Project Aria's VRS carries per-record timestamps with explicit time domains —
DEVICE_TIME recommended, HOST_TIME warned against — and TICSync for
cross-device alignment. XDF/LSL carries nominal_srate with 0 meaning
irregular, plus ClockOffset chunks recording measured per-stream drift for
later interpolation. AgiBot World's proprio_stats.h5 carries a top-level
/timestamp in nanoseconds plus per-source indices recording when each control
source actually emitted.
The pattern worth copying for exports is Ego-Exo4D's: a capture-level
timesync.csv with per-camera presentation timestamps and frame numbers,
plus materialised index-aligned media. Naive consumers get correctness;
rigorous consumers can recover real timing. That is what a CA3 export to
LeRobot or Parquet should do rather than adopting the single-fps loss.
§6 — A chunk belongs to exactly one stream
What the spec says. "In a session where video is ninety-five percent of the bytes, reading gaze and inertial data touches roughly two percent of the file. That access pattern is use case 4 — training on a subset of streams, repeatedly, across a corpus."
Measured. measurement/baselines and
crates/container/examples/selective_read.rs confirm the ownership half
exactly: selecting one stream touches that stream's extents and nothing else —
0.01 % of the reference file for trial.events. The ratio in §6's sentence
remains untested because no fixture has a frame stream; §15.1 keeps the row.
Corroborating evidence for why it matters. Mohan et al. (PVLDB 2021) found "DNNs spend 10–70 % of their epoch time on blocking I/O", and Google's fleet analysis found "30 % of the total compute time is spent ingesting data". The bandwidth a GPU actually needs is modest — AIStore measured "~60-70MB/s per GPU" falling to "40-50MB/s" past eight — so the win from reading 2 % rather than 100 % of a file is in not evicting the cache, more than in raw throughput.
And a finding that favours addressability over shards, which our own benchmark did not measure and could: Mohan et al. report "TFRecord format results in 40 % higher cache misses than the ideal because, the sequential access nature of TFRecords (and RecordIO) is at odds with LRU cache replacement policy."
§6.5 and §7.5 — Addressable samples
Argued in full in access-patterns.md. The short version:
- Per-sample uniform random access is not what production training does. WebDataset, MosaicML Streaming, Megatron-Energon and FFCV each abandoned it, and FFCV gives the physical reason: "each file system has a block size after which random reads become sequential reads. In almost all system, this lies below 2MB."
- But shuffle buffers fail on sequentially-recorded scientific data, which is what CA3 holds. scDataset measured a 16,384-sample buffer producing near-chance accuracy where true random sampling worked.
- The convergent answer is block sampling with batched fetching — FFCV's 8 MB pages, scDataset's block 16 / fetch 256, CorgiPile's 2 % buffer — and §6.5's real property is that a contiguous run is one read at a computed offset. That is the target, and it is not the one we first measured.
§0.6 — A file defines itself
What the spec says. "Anything a reader needs in order to interpret a recording belongs inside the recording, not in a registry, a lookup table, or a copy of this repository."
Independent corroboration, which is the strongest kind. NWB reached the
same conclusion from a different direction: its HDF5 storage specification
reserves a /specifications group holding the schema a file was written under,
extensions included, "so that users can access the specification directly if
necessary without requiring external resources." A stranger with PyNWB opens a
ten-year-old file with nothing installed.
And the counter-example is inside CA3's own history. §5.7 retired
schema_conformance = 2 because it "was the single value in this format that
a reader could not settle from the bytes in front of it" — it asked a reader
to consult a registry that might not be reachable. Two projects, the same
principle, one of them arriving at it by deleting a feature.
Zero dependencies, and no C library
What the workspace says. crates/container has no dependencies, and
unsafe_code = "deny".
What that is worth, measured. Building an HDF5 baseline for
measurement/baselines required either hdf5-metno, which needs a
discoverable libhdf5 or CMake plus a C toolchain, or hdf5-pure, which needs
neither and lacks the selection API the comparison required. Neither route
produced a number on the machine that ran the benchmark, on which cmake,
HDF5_DIR, hdf5.dll and cl.exe are all absent. The measurement had to go
through Python instead.
That is the cost a C-linked container imposes on every consumer, met rather than theorised — and it is the reason a browser-side reader is possible for CA3 and not for HDF5.
Immutability, and why concurrency is not a problem here
What the spec says. §2.3: "Every record is appended in final form and nothing is back-patched." A sealed file never changes.
What that avoids. HDF5 is thread-safe by serialising:
H5_allow_concurrent_g is FALSE in the 2.0 documentation, and h5py's own
threading page states that multiple API calls do not run in parallel even on
different files. A fork()ed child inherits library state and file
descriptors, which is why a PyTorch DataLoader with num_workers > 1 must
open the file inside __getitem__ — an issue open since 2018. The
multi-threaded HDF5 effort has been an RFC since 2021 and is not in 2.0, 2.1 or
2.2.
The cause is mutable global state: a metadata cache, an open-object registry,
free-space tracking. CA3 has none of it on the read path — the only Mutex
in crates/container is in recorder.rs, the writing path — because a sealed
file is immutable and §7.1's index is strided directly with no parsing.
What CA3 does not have, stated. No concurrent-writer story: §2.3's model is one writer appending, where HDF5 offers PHDF5 over MPI. And no live-read story: §8 recovers a truncated file, but a reader racing an appender has no defined semantics, where HDF5 offers SWMR. §8's checkpoint chain — a self-locating trailer with a CRC and a backward pointer — is the obvious basis for one and nobody has specified the reader side.
What this document is not
Not a claim that CA3 is better than the formats named. Each was built for a different job: NWB for archive deposition and neuroscience interoperability, Parquet for analytics at rest, MCAP for write throughput on a robot, HDF5 for n-dimensional arrays in simulation. Where those are the job, they are the answer, and a CA3 export to them is worth building.
What the evidence supports is narrower and sufficient: the problems §5.4, §5.5, §5.6 and §10 address are real, are measured, and are unsolved in every format a recording would otherwise be written to. That is why the container exists.