0009 — The format has a write form and a read form, and the converter is the bridge

• Status: Accepted. This record names a structure the format already had and had never stated, which is why it changes no bytes.

• Implementation status: Both forms are built; neither is declared. §6.2 and §6.5 exist and ca3 convert exists. What does not exist is the Header.profile value that lets a file say which form it is, or a converter feature that emits §6.5. Both are named in "What this obliges" below.

  Re-derive the two forms' costs:

  cargo run --release --manifest-path measurement/baselines/Cargo.toml

• Date: 2026-09-09

• Depends on: 0003 — this record is that principle carried to its conclusion, and 0003's "can the converter do this instead?" is the question it answers structurally.

Context

A recording format and a training format want opposite things, and CA3 has been quietly serving both without saying so.

What a capture path needs. §2.3 already states it: "Every record is appended in final form and nothing is back-patched … This is what lets a recorder write to a sink that cannot seek." A device is streaming, a queue is filling, and any cost the writer pays is paid while data is arriving. §6.2's layout follows from that — messages back to back, varint lengths, a field_mask that is the union over the chunk, no per-message framing.

What a training consumer needs. Addressable samples, fixed strides, contiguous runs, and no parse between the file and the tensor. §6.5's layout follows from that — fixed-width columns, a block region, sample i at block_bytes × i.

These are not a trade-off to be balanced. They are two artifacts. A varint cannot be indexed, so density and addressability cannot both be had in one layout; §6.5 says so directly.

MCAP reached the same conclusion from the other side and wrote it down first. Its evaluation document (Foxglove, 2021-10-06) opens:

"A recording format should be optimized for write throughput first and foremost."

Its published benchmarks measure write throughput and contain no random-access measurement at all, which is not an omission — it is the design being consistent with itself.

Decision

CA3 has two forms of the same format, and a file declares which it is.

	Written by	Optimised for	Layout
capture	a recorder, on the path	write throughput, and a sink that cannot seek	§6.2 chunks
derived	a converter, offline	read throughput, and addressability	§6.5 columnar chunks, §7.5 offsets

Same magic, same framing, same records, same self-description. A derived file is a CA3 file that every existing reader opens; it is not a second format and it does not get a second specification. What differs is which chunk record carries the messages and which indexes are present.

Header.profile is where a file says which, and §4 already defines that field as "a named set of conventions". The two values are capture and derived.

ca3 convert is the only bridge, and it runs offline on a finished file. That is 0003 exactly: the cost of the derived form is paid after recording, by whoever wanted it.

What each form promises

capture promises nothing beyond version 1. That is the point — the floor for writing a conformant CA3 file does not move, which 0003 named as a rule: "The bar for writing a conformant CA3 file does not move because a training pipeline would prefer different bytes."

derived promises what a loader may assume without probing. Per §11.4's rule that "A profile MUST state, per assertion, whether anything enforces it":

Assertion	Enforced by
Every stream with a §5.8 block form carries its messages in ColumnarChunk records	the converter; checkable by a reader from the opcodes present
required_features bit 2 is set	§6.5, and a reader refuses the file by name otherwise
Chunks are uncompressed, so block_at addresses without decompressing	checkable: codec == 0
MessageOffsets is present for every §6.2 chunk that remains	checkable: one record per chunk offset
Provenance names the source file, its length and its CRC	already written by ca3 convert
Raw timestamps survive	§0.3's G3, unchanged. A derived file does not resample, and does not replace a measured time with an aligned one

That last row is not decoration. It is the line between a derived CA3 file and an export: an export may project onto whatever a target format can hold, and a derived CA3 file may not, because it is still a recording.

What this is not

Not a new file extension with its own semantics. A .ca3 file is a .ca3 file. A filename convention such as .derived.ca3 is fine as a human signal and carries no meaning a reader may rely on; the declaration is in the header.

Not permission for the derived form to lose information. Re-chunking, re-laying-out, dropping checkpoints and dropping compression are all permitted — the capture file is the archival copy and keeps its recovery properties. Resampling, collapsing clock domains, or discarding a validity channel are not, and would make the output an export rather than a derived recording.

Not a claim that the capture form is slow. It is untested. §6.2 was designed for write throughput and has never been measured against anything, which is the first thing "What this obliges" asks for.

What this obliges

1. Measure the write axis. §6.2 against MCAP, both append-only, both write-first, on the same messages. That is the fair comparison MCAP deserves and it has not been run. §6.2 may lose, and recording that is worth more than not looking.
2. Measure the read axis honestly. §6.5 against a tuned Parquet, with the take rate swept across orders of magnitude and block sampling rather than uniform per-sample access — measurement/methodology.md has the list of what the current measurement gets wrong.
3. Emit the derived form from ca3 convert. Both records exist and neither is written by anything but a benchmark.
4. Define the two Header.profile values in the specification, with the assertion table above, and state what enforces each.

What was rejected

One layout, tuned for a compromise between the two. Rejected because the compromise does not exist: a varint is either indexable or compact. Attempting it produces a format that is second-best at capture and second-best at training, which is the shape of every format that tried.

A second binary format for training. Rejected in 0008 and rejected again here for a further reason: a derived file is still a recording, and everything that makes a CA3 file worth keeping — self-description, declared units and frames, clock domains with their uncertainty — is exactly what a second format would have to re-earn. The profile mechanism costs one string.

Making the capture form promise anything. Rejected on 0003's rule. A producer that only wants to record data must not be asked to buffer, seek, index, or choose a layout while a device is streaming at it.

What would change this

• A capture path for which the derived form is free. A producer already writing from page-aligned buffers with the whole session resident could emit §6.5 directly. §5.9's writer route is the precedent — the principle is that the writer must not be required to, not that it must not be able to.
• A measurement showing §6.2 is not competitive on write throughput. Then the capture form needs work, and this record's premise — that CA3 already had both halves — is half wrong.
• A consumer that needs the derived form on the capture path, for a live training loop reading a session as it is recorded. Nothing in CA3 serves that today: §8 recovers a truncated file but a reader racing an appender has no defined semantics. That is a real gap and it is named in measurement/motivations.md.