0008 — A training layout is built here, and measured before it is chosen

• Status: Accepted. The comparison is the gate, not this record — §15 requires a measurement before ColumnarChunk can be unreserved, and this record says what gets measured against what.

• Implementation status: Built and measured. ColumnarChunk (0x0E) is §6.5 and MessageOffsets (0x49) is §7.5; both are out of §15's reserved table. Sample-index addressing needed no new record at all. Hyperslab is ColumnarHeader::hyperslab, and it is arithmetic rather than a primitive because §5.8 already fixes the element type and a row-major layout.

  What is not built: neither record is written by ca3 convert yet, so a recording gains the layout only through measurement/baselines. That is the next step and it is a converter feature, not a format question.

  Re-derive the comparison:

  cargo run --release --manifest-path measurement/baselines/Cargo.toml

  Re-derive what addressing costs today:

  cargo run --release -p ca3-cli -- mock bench.ca3
  cargo run --release -p ca3 --example index_addressing -- bench.ca3 40000

  Re-derive what a selective read costs, which is the property a training loader is buying:

  cargo run --release -p ca3 --example selective_read -- bench.ca3

• Date: 2026-09-09

Context

A training pipeline asks a recording for arrays: a batch of samples, in a known shape, with a known element type, fast and repeatedly, across a corpus. §0.3's G7 answers that with convertibility rather than direct trainability — "a converter to a columnar training format is mechanical rather than a script encoding undocumented assumptions."

HDF5, Zarr, NWB and BIDS are where a recording goes afterwards, and this record does not weigh CA3 against any of them. They are export targets, and exporting to them is a converter's job rather than this container's — §0.4 already places the dataset layer that way: "Dataset organisation — subjects, sessions, directory layout … Established conventions cover this. A CA3 file is a file within such a dataset." A recording that never reaches BIDS or NWB is a recording that did not get shared.

So the question here is narrower, and it is only about the training loop: does the array layout a training pipeline reads live inside CA3, or only in what an export produces?

§15 has reserved space for the first answer since version 1 — ColumnarChunk (0x0E, feature bit 2) and MessageOffsets (0x49) — and never spent it. The second answer needs no format work at all, which is what makes it the default worth arguing against rather than the option nobody considered.

What an export already gives a training pipeline

This is not a close-run thing on the array side, and pretending otherwise would make the rest of this record untrustworthy. If exporting were enough, 0x0E should stay reserved.

• N-dimensional datasets with explicit dtypes, chunking and compression, solved and tuned for decades.
• Hyperslab reads — a strided rectangular selection out of one large dataset, [100:200, 3:5], computed by the library. CA3 has no equivalent and §9 defines no primitive that could become one.
• Loaders that already exist: h5py, MATLAB natively, C, C++, Julia. A researcher opening one needs nothing from this project.
• In this field specifically, an incumbent. NWB is built on HDF5, so a neuroscience-adjacent consumer already has the tooling installed.

Every one of those is a reason to build the exports, and they are being built regardless of what this record decides. Nothing below argues against them.

What every export has to drop, and this is the whole of the argument

An export is a projection, and what a projection loses is a fact worth stating rather than discovering. This section is not a case against exporting. It is the list of decisions an export plugin has to make, and the reason the training loop is the one consumer that should not have to live with the answers.

§5.2's floor asks a schema for a unit, a coordinate frame with its handedness, the cardinality and ordering of anything repeated, and absence in its own validity channel. §10 adds a clock domain per stream and an error model with its provenance. §5.8 adds a declared tensor shape, element type and axis order.

None of that has a standard representation in HDF5. Attributes can carry it, and nothing says what to write. So an export plugin must do one of three things:

Define an attribute convention	Which is a schema layered on HDF5 — which is what NWB is. The wheel gets reinvented, smaller and privately
Adopt NWB	Correct where NWB's neurodata types cover the data, which for the physiological half they do. The next section is what happens to the XR half
Drop the declarations	Which betrays G7 exactly: the assumptions stop being documented, one step further downstream, where nobody is looking for them

The third is the one that happens by default, because it is the one that ships soonest and the loss is invisible until somebody asks what unit a column is in.

A plugin that projects deliberately and says what it dropped is a good plugin. A training loop that reads through such a projection on every epoch is paying for a loss it never needed to take, because the declarations were in the file it started from.

What an NWB export will have to decide

NWB is a planned export target, not an alternative weighed here — a versioned schema over HDF5 with a validator, an archive, and files that cache their own schema so a stranger can open one in ten years with nothing installed. This section is a survey of what the plugin that writes it will have to answer, recorded now because the answers are design decisions rather than implementation details, and because they are the same questions a BIDS sidecar, an HDF5 export and a Zarr store each face in their own dialect.

It covers the physiological half well. Surveyed at core schema 2.11.0 (2026-08-21):

EEG	ElectricalSeries with ElectrodesTable, a DynamicTable that takes custom columns without an extension. Mature and widely tooled
Pupil	PupilTracking
Discrete events	EventsTable, new in 2.10.0 (2026-06-18), one event per row with timestamp required and a MeaningsTable for categorical values. A good fit for trial markers and button presses
Units	unit, with conversion and offset so stored × conversion + offset yields the declared unit. A better design than most ad-hoc conventions, and worth learning from
Joint positions	ndx-pose 0.3.0, a catalogued extension. Single-subject by its own documentation, and defaulting to pixels rather than metres

Its self-description is genuine and is the strongest thing about it. The HDF5 storage spec reserves /specifications, so a file carries the schema it was written under, extensions included. That is the same instinct as §0.6's "a file defines itself", arrived at independently, and it is why NWB is the serious comparison rather than raw HDF5.

And the XR half has no types at all. Not thin types — absent ones: six-degree-of-freedom orientation (SpatialSeries is position only, CompassDirection is a scalar heading), IMU, hand tracking, controller analog axes, controller pose, pointer and raycast targets. The NDX catalogue carries no XR, headset or immersive-environment extension.

Three of them are not gaps but opposite commitments, and those are the ones that decide this:

CA3 requires	NWB provides
§5.5 — a frame declares its origin, axis directions and handedness, because "camera space is not a frame" and a renderer's handedness once got written into a recording that outlived the renderer	SpatialSeries.reference_frame, a free-text string. No handedness field, no axis order, no transform graph
§5.6 — absence travels in a validity channel, never as a sentinel, because "a model trained on that learns the origin is a place people look"	Best practice encourages NaN as the missing-data sentinel. invalid_times is file-level and interval-granular; control is one uint8 per timepoint for a whole series, not per channel
§10 — every stream carries its own clock domain, and a timing claim carries its uncertainty and provenance	One time zero per file. timestamps_reference_time is "time zero of all timestamps", and the sync group is documented as "lab-specific… as provided directly from hardware devices" — deliberately unstandardised

The third is the sharpest. NWB's model asks you to resolve every clock to one master timeline before writing, and to discard the evidence of how you did it. §0.3's G2 and G3 exist to refuse exactly that: "raw timestamps survive to the reader", and a reader "receives a time and an uncertainty, decomposed into the terms that produced it, each with its provenance."

So the NWB plugin has real design work ahead of it, and this is the shape of it. It will define an ndx- extension for the XR half, and it will have to state — per stream, in the export's own provenance — how it resolved the three above: which frame convention it assumed, what it did with a validity channel NWB has no column for, and which clock it collapsed everything onto. A plugin that records those three answers is doing its job. One that does not is producing a file whose numbers are right and whose meaning is a guess.

What the export buys is distribution, and that is worth having. DANDI deposition and neuroscience interoperability are things nothing in this repository can offer, and the extension work is the price. None of it is an argument about what the recording itself should be — only about what has to survive the projection, and what cannot.

BIDS is the other standard in this space, and its eye-tracking support is 2D. BEP020 merged 2026-01-16 and shipped in BIDS v1.11.0 on 2026-02-04, as a specialisation of physiological recordings: timestamp, x_coordinate, y_coordinate, with RecordedEye and SampleCoordinateSystem required. That is a gaze point — on a screen, in the head, or in the world. A 3D eye model — an eyeball centre as a position and a gaze direction as a vector, each in a frame that declares its handedness — has no home in it. Which is the same gap §5.5 was written from, and §5.5 names the incident: "an integration once negated the y component of an eyeball centre because the convention was copied from a rendering package."

Two practical notes for anyone who reaches for NWB anyway. Rust support is effectively nonexistent — no library API; the recommended writers are PyNWB, MatNWB and AqNWB. And the Zarr backend is beta and pinned to Zarr v2 (hdmf-zarr 0.13.0), so NWB means HDF5 in practice.

What CA3 already has

Measured or read out of the specification, not assumed:

Declared array shape	§5.8 block tensor forms fix the element type, the byte order, the axis order and each component's unit. Row-major always, so a consumer cannot silently transpose
Zero-copy addressing	§5.9 puts a stream's tensor blocks on a declared power-of-two boundary up to a page, against a pinned or RDMA buffer. Built, and ca3 convert --align applies it to a finished file
Selective-stream reads	§6's keystone. Measured: selecting one stream touches that stream's extents and nothing else — 0.01 % of the reference file for trial.events, 42.98 % for sensor.motion
A window primitive built for this	§9.2 is named "the training and analysis primitive" and carries three MUSTs, including that the same file and window produce the same sequence every time
Sample-index addressing	Works today on a version 1 file, and needs no new record

That last row was the surprise, and it is worth stating precisely because it reduces the work.

§9 has no primitive keyed on sample index — §9.1 asks what a stream held at an instant and §9.2 asks for a window, and a shuffled loader asks neither. But ChunkIndex (§7.1) already carries message_count per chunk, §7.1 already sorts one stream's entries contiguously, and StreamDirectory (§7.3) carries the stream's total. A running sum over the first is monotonic, so a binary search turns a sample index into a chunk — and the directory's own figure is a second statement in the file to check the sum against rather than checking it against itself.

On the reference recording every stream agrees, and sample 40,000 of sensor.motion resolves to chunk 18 at offset 2,872,886, then a walk of 670 records. That walk is the cost, and it is what MessageOffsets exists to remove.

The head-to-head

100,000 samples of a [28, 8] f32 tensor — a body-tracking stream's shape, 85.4 MiB — with 1,000 samples per chunk or row group, so the comparison is of layouts rather than of chunk sizing. Compression off everywhere. All six checksums agree, and the checksum is index-weighted, so a reader returning the right blocks in the wrong order disagrees rather than passes.

target	bytes	scan	random	slab	random reads
ca3 columnar mmap	90,409,000	18.6 ms	1.2 ms	1.2 ms	37.6 MiB
ca3 columnar batched	90,409,000	—	13.6 ms	n/a	14.7 MiB
ca3 columnar (§6.5)	90,409,000	0.6 ms	29.0 ms	0.8 ms	8.5 MiB
ca3 row (§6.2)	90,306,600	27.1 ms	4,035.2 ms	n/a	8,612.2 MiB
mcap	94,313,487	16.2 ms	16.2 ms	n/a	89.9 MiB
parquet	90,586,100	18.6 ms	2,621.0 ms	n/a	8,633.7 MiB

§15's trigger is met. It asked for "a measurement showing §6.2's row-wise encoding costs materially more": 1,013× in bytes fetched and 137× in warm time, on the access pattern a shuffled loader uses.

random reads is the column that survives scrutiny, because bytes fetched do not depend on this machine's memory. It exists because a file larger than page cache could not be written here — 63.8 GiB of RAM against 38.5 GiB of free disk — so the cold case is derived from bytes rather than left unmeasured. Cold, time follows that column.

Where CA3 does not win, stated. MCAP's warm random access (16.2 ms) beats plain columnar (29.0 ms), because MCAP streams sequentially while columnar issues ten thousand seek and read pairs. It fetches ten times the bytes to do it, so the ordering reverses cold — and mmap closes it warm at 1.2 ms — but as measured, that cell is MCAP's.

Two mistakes were caught before any of this was quotable, and both had flattered CA3. The Parquet target opened the file inside the random loop, ten thousand open() calls, and reported 6.9 s as a property of Parquet. MCAP's writer defaults to zstd, which put its file below the raw tensor size and made its scan column a zstd measurement. A third is stated rather than fixed: scan and random measure reaching each sample, not copying it out, so a loader that copies the whole block pays the same memcpy in every row.

And the construction caveat. §6.5 was designed for this access pattern and then measured on it. Parity was the stated goal, so that is legitimate, but it is not an independent benchmark and should not be quoted as one. A Parquet layout with 224 exploded Float32 columns would score on slab and fetch far less per random read — at the cost of a 224-column schema and a naming convention invented on top, which is the trade this record exists to notice.

HDF5 is absent from that table, and why is a finding

Three routes were tried on the machine that ran this.

hdf5-metno, the maintained C binding, would have given the most comparable numbers. It links libhdf5: a discoverable system copy, or the static feature, which makes CMake a hard build prerequisite. cmake, HDF5_DIR, hdf5.dll and cl.exe are all absent here and rustc is MSVC-hosted, so neither route builds.

hdf5-pure 0.44.1 is pure Rust, builds in seconds with no C toolchain, and compiles to wasm — genuinely notable. But at that version Dataset exposes read_f32() and no selection API: no hyperslab, no partial read. A row built on it would have shown HDF5 losing random access by four orders of magnitude, and that number would have described the crate rather than the format.

measurement/baselines/h5py_baseline.py is the route that survives: the consumer a researcher actually uses, whose wheels bundle libhdf5 and need no compiler. It is written and refuses to guess when h5py is absent.

What is missing, and it is one thing

1. ColumnarChunk (0x0E). §6.2 puts messages back to back, each with its own length and stamp delta, so a batch of n samples is n parses. A columnar layout makes the blocks contiguous and a batch one read.
2. MessageOffsets (0x49). The 670-record walk above. §15 is explicit that for a compressed chunk this saves parse work and no I/O, and for an uncompressed one it saves both. Both records are built. What remains is the converter feature that emits them, so a recording can be prepared for training with ca3 convert rather than by a benchmark harness. That is decision 0003's shape and needs no format work.

One measurement worth carrying forward, because the intuitive guess is wrong: §5.9 page alignment would make the mmap case worse at this block size. A page is 4.6× an 896-byte block and shuffled samples rarely land adjacent, so almost every block already costs a whole page — 37.6 MiB fetched to deliver 8.5, a 4.4× amplification. Aligning gives the same fault count and a file 4.6× larger. §5.9 says why in its own text: the boundary's cost is inversely proportional to the payload. Page alignment is for a frame stream.

Decision

Build the three inside CA3, and measure them head to head against a general array container before treating the question as settled.

§15's trigger for ColumnarChunk already demands this — "a dense multi-channel stream, plus a measurement showing §6.2's row-wise encoding costs materially more." A reserved opcode is not spent on an expectation.

So an HDF5 export is not the loser in this record; it is the baseline. Writing the same recording both ways and measuring the same access patterns is the only way to make the claim rather than assert it, and the export is worth keeping afterwards for consumers who live in h5py and MATLAB either way.

The export core is built with a pluggable sink, per 0005's representation table, which is sink-independent: how a nested message becomes names, how validity_for becomes a mask, how a tensor form becomes an array. Those answers are identical whether the output is CSV, HDF5, or a CA3 columnar chunk. Getting them right once means a format choice costs a sink rather than the semantics.

What was rejected

Treating export as the only answer, and building nothing here. The exports are being built either way; the question was whether the training loop should read through one. Rejected because a projection that a plugin performs once per corpus is cheap, and one a loader performs on every epoch is not — and because the declarations the projection drops are exactly the ones a training consumer needs. Not rejected permanently: the measurement can still show the columnar layout is not worth 0x0E, and then export is the whole answer.

A separate .ca3.train binary format. Rejected because it buys nothing the container does not already offer and costs a second specification, a second conformance suite and a second reader in every language. A training file is a CA3 file: same magic, same framing, Header.profile naming the conventions it satisfies, and required_features naming what it uses. Every existing tool opens it, and §0.6's principle that a file defines itself survives intact.

Building ColumnarChunk before the measurement. Rejected by §15's own rule. The measurement comes first even though the outcome is expected, because an expectation that was never tested is how a reserved opcode gets spent on the wrong shape.

What would change this

• An independent benchmark contradicting the table above. The measurement is self-built, on a self-designed layout, with an access pattern chosen by the same author. That is the weakest thing about this record. MLPerf Storage and DLIO exist; running against one, or against a real PyTorch DataLoader, would replace the caveat with a result.
• Shuffled random access turning out not to be the pattern that matters. WebDataset gives it up deliberately — sequential shard reads with a shuffle buffer — and so does MosaicML Streaming. If that is what large pipelines actually do, a layout optimised for random indexed access is optimising a pattern the field has routed around, and MCAP's sequential number is the relevant one. This is the objection most likely to overturn the record.
• An XR vocabulary arriving in NWB. The section above is a survey of core schema 2.11.0 and the NDX catalogue on 2026-09-09, not a permanent claim. A catalogued extension covering 6-DOF pose, hand tracking and controller input — with declared frames rather than prose ones — would make exporting to NWB reuse a convention rather than invent one, and this record would be worth reopening. The clock-domain and validity-sentinel oppositions would remain.
• Archive deposition becoming a requirement. If recordings must reach DANDI, NWB export stops being optional and its constraints are simply the cost of distribution. That changes what is also built; it does not change what the recording is.
• A consumer population that lives in one array container. The strongest argument against this record is social rather than technical: a format nobody's tooling opens is worse than a lossy export everybody's does.
• Hyperslab turning out to be a real access pattern here rather than a capability that exists because HDF5 has it. If sub-window reads across a tensor's dimensions are what somebody actually does, that is a primitive worth designing rather than a parity item.

Notes on the Rust HDF5 situation, since a baseline has to be built

Recorded because it decides how the baseline is built, not to argue against HDF5. Gathered 2026-09-09 from published sources and not verified by building anything. The same caveat covers the NWB survey above: it is a read of the schema and the catalogue, not of files written by either.

• The crate everybody names is not the maintained one. hdf5 has been frozen at 0.8.1 since 2021-11-21. hdf5-metno is the live fork — 0.14.1 on 2026-08-12, releases roughly monthly, 69 reverse dependencies. Write-side coverage is complete for this purpose: n-d datasets with dtypes, chunking, gzip/blosc/szip filters, attributes, compound types, variable-length types.
• The build is the friction, not the API. Either a system libhdf5 that must be discoverable, or the static feature, which makes CMake a hard build prerequisite and compiles the C library. Cross-compilation is a known open problem in that crate.
• There is no WebAssembly path through the C-backed crate, which matters because a web-native reader is a goal here. hdf5-pure is a pure-Rust implementation that does compile to wasm, six months old at the time of writing.
• HDF5 is thread-safe by serialising. H5_allow_concurrent_g is FALSE in the 2.0 documentation, and h5py's own threading page states that multiple API calls do not run in parallel even on different files. Reading one file from forked worker processes remains a documented sharp edge.

So the baseline is built behind a non-default feature in a separate crate, and crates/container keeps the zero dependencies it has today.