The container format, version 1
Status: draft. Nothing here is frozen.
This document is the contract. The implementation in crates/ is one producer
of files this document describes, and where the two disagree, one of them is
wrong and both get fixed — a divergence is not a difference of opinion.
The test this document is written to pass: a third party implements a reader from it, with no access to this repository's source.
0. Scope, goals, and non-goals
0.1 What CA3 is
A container for one session of timestamped measurements from multiple instruments, and the vocabulary that fixes what those measurements mean.
Three parts, and the third is what makes the first two worth having:
| The container | How records are framed, how messages are packed into chunks, how the index is built, and what a reader does with a file whose writer was killed |
| The vocabulary | What a measurement is — units, coordinate frame, ordering, validity — and the array shape it projects onto |
| The time model | Which clock every timestamp is on, how domains relate, and how well that relationship is known |
CA3 is not tied to any runtime. A file written by a dataflow system, a conversion from an existing archive, and an analysis step emitting derived streams are the same kind of artifact, and every CA3 tool treats them identically.
0.2 Use cases this format is designed against
Each goal below traces to one of these. A proposed feature that serves none of them is out of scope, whatever its merit.
- A regulated study. Data collected for a regulatory submission, where what produced a record, under whose authority, and on which clock are questions asked of the file rather than of the people who ran it.
- A multi-instrument session. Several devices, each with its own clock, where the questions of interest live in the 10–100 ms range and a 36 ms unrecorded offset changes the answer.
- Comparison across studies, laboratories, and vendors. The same measurement means the same thing in a file someone else wrote.
- Training data, and model output. A recording becomes arrays without a human deciding what the numbers meant; model output stores in the same vocabulary it was trained on.
- Producers other than the reference runtime. Converters, other acquisition systems, and analysis steps writing derived streams.
0.3 Goals
Ordered by how expensive they are to get wrong. The first two are irreversible at the first recording anyone intends to keep.
G1 — A measurement means the same thing in every file that carries it, when the schema says enough for that to be true.
A schema that clears §5.2's floor carries units, a coordinate frame with its handedness, the cardinality and ordering of anything repeated, and absence in its own validity channel rather than as a sentinel value. Schemas beyond the canonical set are first-class; what is required of all of them is that they are defined, and a file states which of its streams clear the higher bar.
Serves: 3, 4. Test: two producers, different vendors, same modality — one reader consumes both with no per-file adaptation.
G2 — Every timing claim carries its own confidence, computed one way everywhere.
A reader receives a time and an uncertainty, decomposed into the terms that produced it, each with its provenance. The arithmetic is normative, so two producers observing the same event report the same number. Unknown is a conformant answer and a visible one; silently absent is neither.
Serves: 1, 2. Test: a reader returns a time with a stated basis and bound for every sample; a file with no alignment evidence says so rather than resembling one that has it.
G3 — Nothing derived is presented as measured.
Raw timestamps survive to the reader. A reader may return aligned time and must label it. The chunk index is keyed on each stream's own raw domain-local time, so a better estimate changes the transform applied at query and never a byte of the file.
Serves: 1, 2.
G4 — A file that cannot be fully understood is refused, not partly read.
A record whose absence would change what the file says cannot be silently skipped. Three mechanisms are needed and one does not substitute for another: the opcode partition catches an unknown record, the required-feature bitmap catches an unknown value inside a known record, and a data-section walk catches a record a summary-driven reader would never parse.
Serves: all.
G5 — A recording survives its writer, its runtime, and its authors.
Schemas travel inside the file and are restated periodically, so a truncated file stays interpretable. A recording decodes from this document plus a protobuf implementation, and needs neither this repository's code nor any runtime.
Serves: all.
G6 — Anyone can produce a conformant file without adopting a runtime.
Test: a converter written from this document alone, by someone who has not read the reference implementation.
G7 — A recording converts to training data without a human deciding what the numbers mean.
The claim is unambiguous convertibility, not direct trainability: the shape, units, frame, and validity are declared, so a converter to a columnar training format is mechanical rather than a script encoding undocumented assumptions.
Serves: 4.
0.4 Non-goals
Stated so that this list is a specification and not a wish list. Each names what covers the ground instead.
| Not a goal | Because |
|---|---|
| Guaranteeing alignment accuracy | Accuracy is produced by hardware and procedure at capture time. This format guarantees that it is stated, computed uniformly, and checkable |
| Competing with hardware synchronisation | A file carries a PTP or trigger-derived mapping as an ordinary method with a tight bound. It does not try to improve on one |
| Real-time transport | A separate concern with separate systems |
| Dataset organisation — subjects, sessions, directory layout | Established conventions cover this. A CA3 file is a file within such a dataset |
| Selective erasure inside a file | One file holds one participant's session; withdrawal is a delete. Where a subset must be retained, re-deriving a new file with recorded provenance is correct on media where overwriting is not |
| Addressing an individual message inside a chunk | Established containers do this better. Recoverable later; not worth blocking version 1 |
0.5 Conformance language
MUST, MUST NOT, SHOULD, and MAY carry their RFC 2119 meanings.
A requirement on a reader binds anything that interprets a CA3 file. A requirement on a writer binds anything that produces one. Where a rule binds only at a stated conformance level, §16 says so; a rule stated without a level binds every implementation.
An implementation states the conformance level it implements and MUST NOT claim a property of a level it does not implement.
0.6 Stability, and what freezes when
Version 1 names this format's shape. It is not a compatibility promise, and
while this document's status is draft it makes none.
Nothing here is frozen. Field numbers, record layouts, opcode assignments, and the primitive encodings may all change without notice, without deprecation, and without migration. Files written today are not guaranteed readable by anything written tomorrow. A change is an edit, not a version.
This is a decision rather than an absence of discipline, and it has a stated end.
Why. What tells you a layout is wrong is running real sessions through it. Doing that under compatibility rules leaves two options — carry every early mistake forever, or state a guarantee and break it — and both are worse than saying plainly that the guarantee has not started.
The trigger that ends it: the first recording anyone intends to keep. Not a test fixture, not a synthetic session, not a rig capture nobody would miss — a recording somebody would be upset to lose. At that moment, and not before:
- Field numbers become permanent. A recording decodes by number, so once a
file exists in which field 4 meant
landmarks, 4 can never mean anything else for as long as that file must be readable. Field names never become permanent. - The extension rules in §14 start binding, and additive-only evolution begins.
- The major version byte becomes meaningful, and changing it means an incompatible container that readers must refuse.
Until then, treat every layout in this document as provisional and every file as disposable.
What does not wait for the trigger. The shape of the vocabulary — that a unit is declared, that a frame states its handedness, that absence has its own channel — is a design commitment now, because a file written without a place for something cannot have it added retroactively. Numbers may move; the requirement that the field exists may not.
The principle underneath all of this: scope every decision so that changing
your mind later is additive rather than breaking. §14's four mechanisms exist
to do exactly that — an unknown opcode is skipped rather than fatal, a record
grows by appending, a table grows by entry_size, a message grows through
field_mask. Each converts a future decision from breaking to additive, and a
proposal that cannot be made additive by one of them deserves the scrutiny that
a permanent choice earns.
Some decisions are unavoidably one-way once files exist: the primitive
encodings, the magic and the version byte's position, the field order within a
record that already exists, the opcode partition, and what a required_features
bit means. Everything else can wait.
The corollary is that a cheap one-way decision made early prevents an expensive one made late. Reserving an opcode costs a byte and nothing else; not reserving it means a later record must take a byte that shipped readers may already treat as unknown-must-understand and refuse a file over. Reserve early, define late.
And a file defines itself. Anything a reader needs in order to interpret a
recording belongs inside the recording, not in a registry, a lookup table, or a
copy of this repository. Where version 1 falls short of that — §5.5's frame
declarations have no record to travel in — it is a gap to close rather than a
boundary to defend. schema_conformance's retired third value is what happens
when the principle is not held: it asked a reader to consult an authority, and
no format can promise that the authority will be reachable.
0.7 What version 1 does not do
Every deferral appears in §15 with the observable condition that reopens it, and every unmeasured claim appears there marked as unmeasured. A claim that is neither implemented nor listed there is an omission and should be reported.
1. Notation and primitive encodings
1.1 Byte order, and the absence of floating point
Little-endian throughout. Network byte order is big-endian by convention, and nothing here goes on a network. Every platform this targets is little-endian, so big-endian would mean a byte swap on every field of every read on every machine that ever opens one of these files, in exchange for matching a convention about a different problem.
There are no f32 or f64 fields anywhere in the container. Timestamps
are integer nanoseconds and sizes are integer bytes; a float introduces a
rounding question into a structure whose job is to say exactly where something
is.
The clock mapping is where this costs something, because a skew estimate really
is a ratio. It is carried as parts per billion in an i64 — one part per
billion is one nanosecond per second, finer than any estimator will justify —
and the bounds beside it as u64 nanoseconds with documented sentinels.
That is not only rule-keeping. A predecessor format carried its bias bound as
an f64 whose unbounded case was f64::INFINITY, and JSON cannot write
infinity — which is the likely reason that field never reached the session
record where every other figure from the same structure did. An integer with a
documented sentinel survives every serialisation a consumer might use.
1.2 The primitives
| Name | Encoding |
|---|---|
u8 | one byte |
u16 | two bytes, little-endian |
u32 | four bytes, little-endian |
u64 | eight bytes, little-endian |
i64 | eight bytes, little-endian, two's complement |
string | a u32 byte count, then that many bytes of UTF-8 |
bytes | a u32 byte count, then that many raw bytes |
bytes64 | a u64 byte count, then that many raw bytes |
map | a u32 entry count, then each entry as two strings |
array<T> | a u32 element count, then that many T |
raw | bytes with no prefix, extent known from elsewhere |
varint | unsigned LEB128 |
zigzag | a signed value mapped onto a varint |
An array<T>'s elements are fixed-width only when T is. Where an
element is itself a record's content — array<bytes> in §8.1 — each element
carries its own bytes length prefix, which is what lets a reader step from
one to the next without parsing either.
string MUST be valid UTF-8. A reader MUST reject a string field
whose bytes are not, rather than substituting replacement characters: a stream
name that silently changes is a stream that cannot be matched across files.
A writer MUST reject a bytes value longer than u32::MAX rather than
truncating it. Truncation writes a shorter value with no error, and for a
schema descriptor that produces a file whose self-description is silently
incomplete. Where a field may exceed four gibibytes it is declared bytes64.
1.3 Maps are ordered
A map's keys MUST be unique and in ascending order by their UTF-8 bytes.
Two files with identical content must produce identical bytes, and an unordered map breaks that before any integrity or reproducibility work begins. A writer that builds a map by insertion sorts and deduplicates it before encoding; where a key appears twice, the later value wins.
A reader MUST reject a map whose keys are unordered or repeated.
1.4 Varints, and where they are allowed
varint: 7 bits of payload per byte, low bits first,
high bit set on every byte but the last.
zigzag maps a signed value onto an unsigned one so that small magnitudes of
either sign encode short:
zigzag_encode(v) = (v << 1) ^ (v >> 63)
zigzag_decode(u) = (u >> 1) ^ -(u & 1)
Two rules a reader MUST enforce:
- No varint may exceed ten bytes, which is enough for any 64-bit value.
- A varint MUST be the shortest encoding of its value. A multi-byte encoding whose final byte is zero says nothing the shorter form did not, and accepting both would make a hash over the bytes depend on which form a writer happened to choose.
Every field of every record outside a chunk is fixed-width. Varints appear only inside a chunk's message stream, where a field repeats once per message and density is worth the decoder.
The split is deliberate and load-bearing for §16: a reader that wants only a file's structure — its streams, its schemas, the extent and time range of every chunk — can be written with no varint decoder at all.
2. File layout
<Magic><Header><Data section>[<Summary section>][<Summary Offset section>]<Footer><Magic>
2.1 Magic
Eight bytes, identical at both ends of the file:
0x89 'C' 'A' '3' 0x31 '\r' '\n' 0x1A
^^^^ ^^^^^^^^^^^^^ ^^^^ ^^^^^^^^^^ ^^^^
| format id major CRLF DOS EOF
| ver. canary
not 7-bit clean
Each byte earns its place. 0x89 trips a transport that strips the high bit.
\r\n catches CRLF mangling by a text-mode copy. 0x1A stops a Windows
console from dumping binary when someone types the file. 0x31 is ASCII 1,
the major version.
Trailing magic present means the file is sealed. Reading the last eight bytes is one seek and one read, so a writer killed mid-session is detected without parsing anything. §8 specifies what a reader does then.
2.2 Sections
Data section — everything between the Header and the summary. Written strictly in append order.
Summary section — a complete, sorted restatement of the index records, so a sealed file is seekable from one tail read. Records MUST be grouped by opcode in ascending order. Grouping is what lets a reader fetch exactly one index kind in one range request, which matters on object storage.
Summary Offset section — one entry per summary group present.
Both optional sections are absent from a truncated file and from a file a writer chose not to seal.
2.3 The writer never seeks
Every record is appended in final form and nothing is back-patched.
This is what lets a recorder write to a sink that cannot seek — an Android content-provider stream is the case that forces it. The cost is paid in §8: a truncated file has no tail, and the checkpoint chain replaces it.
Two consequences that appear later and are stated here because they follow from this rule alone. Any value a writer cannot know until seal MUST be decided at open and MAY therefore be conservative — §3.3's feature bitmap is the case. And any operation needing writes at arbitrary offsets is not a writer operation; it runs offline on a sealed file.
2.4 What a file must contain
The records are specified across §4, §5, §6, §7, §12 and §13. This is the whole list in one place, and what the minimum actually is — the question an implementer asks first and the rest of this document answers only in pieces.
Every record version 1 defines. Opcodes 0x01–0x3F and 0x80–0xBF are
must-understand; 0x40–0x7F and 0xC0–0xFF are skippable (§3.1). §15 holds
the reserved ones nothing writes yet.
| Byte | Record | Where | Required |
|---|---|---|---|
0x01 | Header | first record, after the magic | always |
0x03 | Schema | data section, and restated in the summary | when a stream names one |
0x04 | Stream | data section, and restated in the summary | when the file has a stream |
0x05 | Chunk | data section | when the file has a message |
0x06 | Session | data section, and restated in the summary | no |
0x07 | Checkpoint | data section, periodically | by durability policy (§8.5) |
0x08 | ChunkIndex | summary | in a sealed file |
0x09 | StreamDirectory | summary | in a sealed file |
0x0A | ClockBlock | summary | when a clock relationship was fitted |
0x0E | ColumnarChunk | data section | when a converter laid a stream out for random access (§6.5) |
0x40 | Metadata | data section (§13.3) | no |
0x42 | Attachment | data section (§13.3) | no |
0x44 | Statistics | summary | in a sealed file |
0x45 | Annotations | data section (§12) | no |
0x48 | SummaryOffset | after the summary | in a sealed file, one per group present |
0x49 | MessageOffsets | summary | when a converter indexed a chunk's messages (§7.5) |
0x4A | Padding | anywhere | no |
0x02 | Footer | last record, before the trailing magic | in a sealed file |
That list lives in code as two tables, and both are half of it.
crates/container/src/opcode.rs:78 (Opcode) holds the thirteen records this
repository's reader decodes; crates/container/src/opcode.rs:204
(defined_but_unimplemented_name) holds the three it names but does not read.
Their union is the table above, and reading only the first is what left 0x40,
0x42 and 0x45 off it until 2026-08-23. So the check reads both:
{ sed -n '/^### 2\.4 /,/^## 3\. /p' spec/container.md |
grep -E '^\| `0x' | grep -oE '0x[0-9A-F]{2}' | sort -u
{ grep -oE '= 0x[0-9A-F]{2},' crates/container/src/opcode.rs |
grep -oE '0x[0-9A-F]{2}'
sed -n '/fn defined_but_unimplemented_name/,/^}/p' \
crates/container/src/opcode.rs | grep -oE '0x[0-9A-F]{2}'
} | sort -u
} | sort | uniq -u | tr '\n' ' '
It prints nothing while the document and the two tables agree, and prints the
opcodes that appear in one and not the other when they do not. Run against this
section as it stood on 2026-08-22, it printed 0x40 0x42 0x45.
The order of a sealed file, start to end:
magic 89 43 41 33 31 0D 0A 1A 01
Header required_features at offset 17
data section Chunk, and the definitions as they occur
summary section ascending by opcode: 03 04 06 08 09 0A 44
SummaryOffset ... one per group present
Footer
magic its presence is what "sealed" means
The minimum is smaller than it looks. A sealed file with no streams and no
messages is the magic, a Header, a summary carrying ChunkIndex,
StreamDirectory and Statistics, a SummaryOffset for each of those three, a
Footer, and the magic again. Everything else appears because the recording had
something to put in it: no streams means no Stream records, no fitted clock
means no ClockBlock, and a writer that was never told a session means no
Session.
A summary group is written only when it has content, which is why the list
above is a contract rather than a fixed layout. A reader locates a group through
SummaryOffset and MUST NOT assume a group is present because another one is.
An unsealed file is the magic and a Header, then whatever was written before
the writer stopped. No summary, no footer, no trailing magic. It is a valid
CA3 file and §8 is how it is read: a reader recovers what was durably written
and reports where the walk stopped. A file that ends mid-record is truncated, not
malformed — the distinction §8.3 draws, and the reason a session interrupted by a
power cut is still worth opening.
What a reader may refuse. The leading magic, and only these: bytes that are
not CA3 at all, a major version this reader predates (§2.1), and a
required_features bit it cannot name (§3.3). Everything else it either reads or
skips.
3. Record framing and the opcode partition
Every record in every section is:
<opcode u8><content_length u64><content>
content_length counts the content alone, so a framed record occupies
9 + content_length bytes (crates/container/src/lib.rs:132, FRAME_BYTES).
3.1 The partition
Whether an unknown record may be skipped is decided by its opcode, not by a flag inside it. A flag has to be parsed before it can be trusted, and a reader that does not understand a record is exactly the reader that cannot parse its body. Putting the decision in the opcode means it is answered by the first byte.
| Range | Class | An unknown opcode here |
|---|---|---|
0x00 | invalid | reject |
0x01–0x3F | format, must-understand | reject, naming the opcode |
0x40–0x7F | format, skippable | skip content_length bytes, and report it |
0x80–0xBF | private, must-understand | reject |
0xC0–0xFF | private, skippable | skip, and report it |
The ranges are answered by crates/container/src/opcode.rs:57 (class_of),
which is a total function of the byte and returns a class for all 256 of them —
crates/container/src/opcode.rs:245 sweeps every one. So the table above is
the function's five arms, and the check is that it stays so:
sed -n '/^pub fn class_of/,/^}/p' crates/container/src/opcode.rs |
grep -oE '0x[0-9A-F]{2}(\.\.=0x[0-9A-F]{2})? => [A-Za-z:()]+' | tr '\n' ' '
The disposition column is crates/container/src/opcode.rs:217 (dispatch): an
unknown must-understand opcode is an error naming the byte and its offset, and
an unknown skippable one is Ok(None), meaning skip.
The rule that assigns a record to a class: must-understand if a reader that
ignored it would report something different about the data; skippable if it
only accelerates a query or annotates. Every defined record is asserted against
that rule rather than against the table it produced
(crates/container/src/opcode.rs:338).
Most container formats skip what they do not recognise, and that is right for a format carrying decoration. It is wrong here. Suppose a later version adds a record marking a range of a stream invalidated by a calibration fault. A reader from today that skipped it returns the invalidated samples as though they were good, and nothing in the output says anything is wrong. That is not a degraded answer; it is a confident wrong one.
When it is genuinely unclear which class a new record belongs to, it is must-understand. The failure modes are not symmetric: a reader that stops when it should have continued costs somebody an afternoon, and a reader that continues when it should have stopped costs somebody a result.
3.2 Why there is a private half
0x80–0xFF is for application-specific records that will never be part of this
format (crates/container/src/opcode.rs:71, is_private). It is partitioned
on the same rule as the format's own half, because an application's private
record can be just as load-bearing as a format one.
3.3 The required-feature bitmap
The partition catches a new record. It does not catch a new value inside a record a reader already knows — a compression codec it has never seen, a set bit in a chunk's field mask, an unrecognised estimator.
Those are caught by Header.required_features, a u64 at the fixed absolute
file offset 17.
That 17 is not a constant anyone chose; it falls out of §2.1's eight bytes of
magic and §3's nine bytes of framing. crates/container/src/lib.rs:139
(REQUIRED_FEATURES_OFFSET) writes the sum rather than the total, so the check
does the addition instead of comparing two copies of the same number:
grep -oE 'REQUIRED_FEATURES_OFFSET: u64 = [0-9 +]+' crates/container/src/lib.rs |
sed 's/.*= //' | tr -d ' ' |
awk -F+ '{ total = 0; for (i = 1; i <= NF; i++) total += $i; print total }'
crates/container/src/records.rs:1539 seeks to byte 17 of a header this crate
actually encoded and reads the bitmap back, which is the half a derivation
cannot cover: the arithmetic can be right while the writer puts the field
somewhere else.
A reader that does not implement every set bit MUST refuse to open the file,
and MUST name the bits it does not implement
(crates/container/src/records.rs:196, unsupported_features, which returns
one named refusal per set bit rather than a boolean).
The field is first in the Header so that a reader decides whether it can open a file from a 25-byte read, rather than discovering it three gigabytes in.
Bit assignments appear in §15 alongside the capabilities that claim them. A writer MUST NOT set a bit this document does not assign.
A writer decides the bitmap at open, from its configuration, because §2.3 forbids the backward seek that revising it would need. A writer MAY therefore declare a feature that no record ends up using. The over-declaration is conservative and is permitted: it costs a minimal reader a file it could have read, and the alternative costs the non-seekable sink.
3.4 Reserved records are named, not unknown
An opcode this document reserves but does not define is not reported as unknown. The operator's next step differs: an unknown opcode means a writer from a later version of the format, and a reserved one means a capability this format has named, deferred, and stated a trigger for.
A reader meeting a reserved opcode MUST report it by name, and then apply
its class — reject if must-understand, skip if skippable. §15 lists them, and
crates/container/src/opcode.rs:173 (reserved_name) is that list;
crates/container/src/opcode.rs:392 asserts the answer is a name rather than
"what is that".
A third answer sits beside those two, and §2.4's table is where it comes
from. A record §12 or §13.3 defines and a given build has not implemented
is neither unknown nor reserved: another producer may be emitting one today, so
calling it reserved — "no version 1 writer emits one" — tells an operator the
opposite of what is true. crates/container/src/opcode.rs:204
(defined_but_unimplemented_name) is the third table, and the two must stay
disjoint or a byte would answer to both:
{ sed -n '/fn reserved_name/,/^}/p' crates/container/src/opcode.rs |
grep -oE '^ 0x[0-9A-F]{2}' | grep -oE '0x[0-9A-F]{2}'
sed -n '/fn defined_but_unimplemented_name/,/^}/p' crates/container/src/opcode.rs |
grep -oE '^ 0x[0-9A-F]{2}' | grep -oE '0x[0-9A-F]{2}'
} | sort | uniq -d | tr '\n' ' '
It prints any opcode claimed by both tables, and nothing while they are disjoint.
4. Header (0x01)
Exactly one, immediately after the leading magic.
required_features : u64 // §3.3; at absolute file offset 17
profile : string // a named set of conventions, or ""
library : string // what wrote it, for a bug report years from now
host_id : string // the recording host
host_clock_epoch : string // which boot of that host
monotonic_source : string // the platform call behind the ordering clock
anchor_monotonic_ns : u64 // a monotonic reading …
anchor_wall_unix_ns : u64 // … and a wall reading taken beside it
anchor_wall_source : string // the platform call behind the wall reading
attributes : map
4.1 The file's own clock domain
(host_id, host_clock_epoch) is the file's clock domain, and a clock domain
is the pair — never the host name alone.
A system monotonic clock restarts at an arbitrary point on every boot, so one machine presents two unrelated timelines under one identity. Treating them as one produces alignment that looks correct and is not.
Every timestamp in the file not attributed to a stream — a checkpoint's emit time, a chunk's receipt base — is in this domain.
4.2 The anchor, and why it is a pair
Monotonic time orders; it says nothing about a calendar. The anchor is a monotonic reading and a wall-clock reading taken as close together as the platform allows, which is what places the ordering timeline on a date.
Either half alone is useless. A wall clock with no monotonic partner cannot be related to the timeline the samples are ordered on, and that relation is the entire point.
anchor_wall_unix_ns of zero means no anchor was taken. A reader MUST
treat a file with no anchor as having no calendar time, and MUST NOT
substitute the file's own modification time or any other external clock.
This was anchor_monotonic_ns until 2026-08-11, and it was the wrong half of
the pair. The argument for it was that a monotonic reading of exactly zero is
the instant of boot, which no recorder observes. That is true of a clock counting
from boot and false of one counting from the session — and a session-relative
clock is what a recorder wants, because it makes a file's timestamps mean
something without knowing when the machine started.
The reference recorder has such a clock, documents it, and writes
anchor_monotonic_ns = 0 on every file precisely because zero is the instant
it anchored. Under the old rule every one of those recordings was undatable by a
conformant reader, while carrying a perfectly good wall reading beside the zero.
Wall-clock zero is 1970-01-01, which no capture observes and no recorder writes by accident, so it is the safer sentinel as well as the correct one.
4.3 Both clock sources are recorded, and this is not bookkeeping
monotonic_source and anchor_wall_source name the platform call behind each
reading — clock_gettime(CLOCK_BOOTTIME), QueryPerformanceCounter,
GetSystemTimePreciseAsFileTime, and so on — so a reader can look up its
documented resolution and behaviour rather than infer them from a language
wrapper.
Naming a wrapper would tell a reader nothing. A 100-nanosecond tick and a
nanosecond timespec are different measurements, and the language type that
returns both is the same type.
monotonic_source is the more important of the two, and it is the one every
format omits. On Linux and Android, CLOCK_MONOTONIC stops during system
suspend and CLOCK_BOOTTIME does not. A device that suspends when a
participant takes it off produces, under CLOCK_MONOTONIC, a recording in
which the break is compressed to nothing — every stream internally consistent,
every checksum valid, and no field anywhere recording that time passed. Two
files that differ only in this choice are indistinguishable without the field,
and the difference is not recoverable afterwards.
A writer MUST record the call it actually used. A reader MUST NOT assume a suspend-inclusive clock from the absence of the field.
Naming a language wrapper does not satisfy this. SystemTime::now and
std::time::Instant are the wrappers, not the calls; the reference recorder
resolves each to the platform call its target uses and writes that, saying so
in the string.
monotonic_source was absent from this record until 2026-08-11, so a file
written before then has a Header one field shorter and does not decode under
this layout — a reader reaches the end of host_clock_epoch and takes the
first four bytes of anchor_monotonic_ns as a string length. §0.6 permits this
and states the terms: "Files written today are not guaranteed readable by
anything written tomorrow." It is recorded here so that somebody holding such
a file can tell an outdated layout from a damaged recording, which the error
alone does not.
5. Schema (0x03) and Stream (0x04)
These two records are what make a CA3 file self-describing: a recording decodes and is interpretable with neither this format's implementation nor any runtime installed.
Both appear twice — once in the data section before the first chunk that references them, and once in the summary. Without the summary copy, indexed random access would have to scan from the file start to find a decoder.
5.1 Schema
schema_ref : u32 // >= 1. 0 means "no schema"
schema_id : string // the protobuf fully-qualified name
encoding : string // "protobuf", or "" for an undescribed payload
descriptor : bytes64 // a serialized FileDescriptorSet, closed under imports
One identity field. schema_id is the protobuf fully-qualified message name
and indexes descriptor directly. A predecessor format carried a second name
because its schema identifier moved the version segment to the end and could not
be looked up in a descriptor set without inverting that transform; carrying one
name that is already the key removes both the transform and the conformance rule
that would police it.
What a writer MUST satisfy is here, and it is the whole list. A schema reference of one or greater, not already declared in this file, and a stream referring to it by a reference of its own that is not zero. That is everything the format requires to accept a schema:
schema_ref == 0 rejected — 0 means "no schema" and cannot be declared
schema_ref declared rejected — a reference identifies one schema per file
stream_ref == 0 rejected — 0 is not a stream
Nothing about units, frames, validity channels or tensor forms. Those are §5.2's
floor, they are a tier rather than a gate, and §5.7 is where a file records which
tier each of its streams reached. encoding may even be empty, for a payload
whose bytes this format carries and does not describe.
descriptor MUST be the transitive closure of the described message's imports
and nothing more. Not the whole schema set the writer happened to have
compiled in.
The reason is not size. A descriptor set that describes a video frame to a reader that asked about gaze makes "this is the schema" a false statement, and forces a validating reader to search where it should be able to check.
Comments are part of a descriptor, and a writer chooses whose. protoc
compiles a .proto's comments into source_code_info, so they travel inside
the recording — which is how a reader four years from now learns that a field
is metres per second squared in a right-handed frame rather than only that it
exists. A writer SHOULD keep the source_code_info of the files it wrote.
It is most of what a descriptor costs and it is the format doing its job.
A writer MAY omit source_code_info from a file it did not write. The case
that motivates the permission is google/protobuf/descriptor.proto, which an
annotated schema's closure reaches through the options file of §5.3. Its
comments document FieldDescriptorProto — the type a reader has already parsed
in order to be reading them — so they are the one part of a descriptor that
cannot reach anybody who did not already have it. Its declarations are a
different matter and MUST stay: the options file names it in dependency,
and a consumer building a descriptor pool from the recording alone fails
without it.
What the permission is worth is the whole of that file's source_code_info,
once per schema whose closure reaches it, and §5's opening says a schema
appears twice in a recording — so a writer that takes it pays that cost twice
per schema, and again in every checkpoint that restates its definitions.
Omitting them changes the descriptor bytes, and §5.7's canonical claim is
checked by comparing those bytes. A recording written by a producer that
keeps them is divergent from one written by a producer that does not, for the
same schema — as it already is against any edit to a comment. §0.6 is what
permits that while this document's status is draft; a producer that changes
its mind after §0.6's trigger fires is changing the registered schema.
descriptor is bytes64 rather than bytes because a schema set with large
embedded options has no small bound, and §1.2 forbids silent truncation.
5.2 The well-formedness floor
This is a tier, not a requirement to write. A CA3 file accepts any schema whose descriptor is present and decodable. What §5.7 records, per stream, is whether that schema also clears the floor below — and a stream that does not is recorded, marked, and read back like any other.
The reason is that a container which refuses data loses data. A session that cannot be written because a schema omitted a unit is gone, and no later care recovers it; an unannotated recording can be annotated afterwards, because the descriptor travels inside the file. Annotations are what a recording is worth more with, not what it must have to exist.
So G1 is what this tier buys, rather than a promise the format makes. A
measurement means the same thing in every file that carries it when the schema
clears this floor. A stream at undeclared makes no such claim — and the
difference is a fact the file states rather than a reason to refuse it.
What a producer must satisfy to write at all is in §5.1, and it is structural: a schema reference that is not zero, not declared twice, and a stream reference that is not zero. Nothing about units, frames or validity channels.
A schema is well-formed when all of the following hold:
- A descriptor is present, closed under imports, and
encodingnames a format a reader can decode. - Every field carrying a physical quantity declares its unit.
- Every field carrying a spatial quantity declares its frame, and the frame declares its handedness.
- Anything repeated declares its cardinality and its ordering. Landmark 5 is the same landmark in every file that carries that schema.
- Absence is carried in a validity channel, never as a sentinel value.
- A tensor form, if the schema declares one, names the fields it takes, their order, and the resulting shape. §5.8 is what that means and what a validator checks of it.
- A coded value is carried as an enum whose values the descriptor carries, or is accompanied by a field whose values resolve without an external table.
Clause 7 is a disjunction because the floor asks for a property rather than a
form. KeySample carries hid_usage, physical_key and text — two codes a
published table decides, and one string that needs nothing but UTF-8. A clause
reading every coded value is an enum would refuse a schema that is correctly
built, because a thousand-entry usage page is not an enumeration anybody should
write out. What the floor asks is that at least one route to a value's
meaning is inside the file, not that every route is.
A validator MUST check that these declarations are present. It cannot check that they are true. A schema that declares metres and carries millimetres is well-formed and wrong, and no amount of format design detects that. The floor buys uniformity of statement, which is what makes disagreement visible; it does not buy correctness.
And no implementation of this format performs that check. The floor is defined here because a reader needs to know what the tier in §5.7 means, and it is judged elsewhere — by a validator holding the descriptor the file already carries. That split is deliberate: the annotations travel as protobuf custom options inside the descriptor (§5.3), so any tool in any language can check them with nothing installed and nothing linked. A writer records the tier it claims; a reader that cares about the claim verifies it against a validator of its own choosing, and neither needs this repository's code to do it.
5.3 How the declarations are carried
As protobuf custom field options, not as naming conventions and not in the reference implementation's function names.
An option travels inside the descriptor automatically, so it reaches a Python, MATLAB, or C++ reader that has never seen this repository. A naming convention reaches only a human.
package ca3.options.v1;
import "google/protobuf/descriptor.proto";
extend google.protobuf.FieldOptions {
// A UCUM code. §5.4.
string unit = 50001;
// The identifier of a frame declared per §5.5.
string frame = 50002;
// Names the field whose validity this field carries.
string validity_for = 50003;
// Fixed element count for a repeated or packed field; 0 means variable.
uint32 cardinality = 50004;
}
extend google.protobuf.MessageOptions {
// The tensor form the message projects onto. §5.8.
TensorForm tensor_form = 50010;
}
(Extension field numbers here are provisional under §0.6, like everything else.)
50010 used to carry a bare string, and the shape behind it lived in a
markdown table in the message's leading comment. That reached a foreign
reader — a descriptor is compiled with source info and the comments travel with
it — but it reached one as prose, so every consumer that wanted the shape wrote
its own markdown parser, and three parsers of one table are three ways to
disagree about it. That is the failure this section rejects naming conventions
for, one level up. The number keeps its allocation and changes its type; §0.6 is
what permits that while the status is draft.
5.4 Units are UCUM codes
unit carries a code from the Unified Code for Units of Measure.
UCUM exists for machine-to-machine unit exchange and is already the unit vocabulary beneath DICOM, HL7, LOINC, IEEE, and ISO 11240 — which is to say, beneath the regulated ecosystems a CA3 file will have to be read in. Inventing a unit vocabulary alongside it would mean a mapping table between two representations of one fact, maintained by us, wrong eventually.
A validator MUST reject a unit that is not valid UCUM syntax.
m, mm, s, ns, rad, deg, m/s2, 1 for a dimensionless ratio.
5.5 Frames declare handedness
A spatial field's frame names a frame declaration, and a frame declaration
states its origin, the direction of each axis, and its handedness.
"Camera space" is not a frame. It is a family of frames that disagree about
which way y points, and the disagreement is invisible in the numbers.
This is not hypothetical. An integration once negated the y component of an
eyeball centre because the convention was copied from a rendering package where
that conversion is correct — a renderer's handedness written into a recording
that outlives the renderer.
A schema may name a frame no catalogue defines, in which case that frame's
declaration MUST travel with the file, and a validator MUST reject a
frame that is neither catalogued nor declared.
The canonical frame catalogue does not exist yet, and every canonical
spatial field therefore fails that rule today. body.tracking_space.v1,
image.plane.v1, imu.producer_defined.v1 and the rest name frames nothing
defines, so a conformant validator rejects them — and the reference validator
does, by name, rather than passing them quietly. This is stated here rather
than left to be discovered because a MUST with nothing behind it reads as a
guarantee, and until a catalogue ships this one is a plan. §15.2 carries it.
Version 1 has no record for carrying a frame declaration inside a file.
A FrameCatalogue is a protobuf message with nowhere to ride, so a file that
names an uncatalogued frame cannot today satisfy the rule above from inside
itself; a validator is given the declarations out of band. That is the second
half of the same gap, and it is why the two are listed together.
A frame may be declared producer-defined, which is an honest statement that the geometry has not been established for that device. It is well-formed. It is not comparable across producers, and §5.7 makes that visible rather than leaving a reader to assume otherwise.
5.6 Absence has a channel
A tensor slot always holds a number, and (0, 0, 0) is a legitimate
position. Encoding "not tracked" as zero makes a real measurement and a
missing one the same bytes, and a model trained on that learns the origin is a
place people look.
So a schema carrying optional measurements carries a parallel validity channel,
and validity_for names the field it covers.
A channel is one of two shapes, and which one it is follows from its cardinality:
| Shape | Encoding | What it says |
|---|---|---|
| Per-element | bool or packed bits, one entry per element of the covered field, in the same order | Element i is valid |
| Whole-field | a single bool | The covered field either carries a measurement or carries nothing |
A whole-field channel is only well-formed when it implies in both directions — false means the covered field holds nothing, and true means every element of it is a real measurement. That is the case for an electrode the amplifier could not read: the channel contributes an empty array or a full one, and there is no third state.
Where a flag implies in one direction only, it MUST NOT carry
validity_for. A body tracker's "somebody is present" is the example: false
does mean every joint is invalid, but true does not mean every joint was
observed — a body can be tracked with a foot out of range. Annotating that flag
as the validity channel tells a consumer to read a tracked body as
twenty-eight tracked joints, which is a decoding rule the schema never meant.
Such a flag stays an ordinary field, and the per-element channel, if there is
one, lives wherever the schema puts it.
The two shapes exist because the absence they record is different, and collapsing them would force a schema to choose between annotating a flag that means something else and leaving a real validity channel undeclared.
A whole-field channel cannot express a hole. A single sample lost inside an otherwise good array has nowhere to go: a producer that meets one invalidates the whole field or cuts the block short. That is a real limit of the version 1 shape rather than an oversight, and §15.1 records that nothing checks the "implies in both directions" condition either.
5.7 Stream
stream_ref : u32 // >= 1. 0 means absent
schema_ref : u32 // 0 when the payload is undescribed
topic : string // the name it was published under
node : string // what produced it
origin : u8 // 0 unspecified, 1 raw, 2 derived, 3 ephemeral
source_clock_domain : string // "" when the stream carries no source time
host_id : string
host_clock_epoch : string
operator_reference : string // opaque; means something in the deployment
operator_asserted_by : u8 // §11.3
authorisation_reference : string
authorisation_from_unix_ns : u64
authorisation_until_unix_ns : u64
nominal_rate_millihz : u64 // 0 when the stream is not periodic
schema_conformance : u8 // 0 undeclared, 1 well-formed, 2 retired
flags : u32 // bit 0: payload_entropy_coded
metadata : map
schema_conformance is how a file states how much of itself it describes,
and it is the reason arbitrary schemas can be first-class without weakening G1.
| Value | Means |
|---|---|
0 undeclared | The writer made no claim. A reader treats the payload as opaque bytes |
1 well-formed | The schema meets §5.2. Decodable, self-describing, unambiguous within this producer |
2 retired | Meant "canonical" — see below. A reader MUST treat it as 1 |
Both remaining values are decidable from the file alone, which is the one
thing this format asks of itself. 0 is a statement the writer makes about its
own silence; 1 is a property of the descriptor sitting in the same file, and
anything holding the file can check it.
2 meant registry membership, and that is why it is retired. It said the
schema was the registered one for its concept, so the stream was comparable with
any other file's stream carrying the same schema — and its own paragraph
conceded the problem: a file cannot assert it into being. It was the single
value in this format that a reader could not settle from the bytes in front of
it, and settling it needed a registry that a reader four years from now has no
guarantee of reaching.
The property it existed for survives without it, and improves. Comparability
is decidable from two files: both carry their descriptors, so a reader compares
the descriptors for a schema_id and gets a definite answer — equal means
comparable, unequal means not, and the difference is inspectable rather than
asserted. That is a fact two files establish between themselves, where 2 was a
claim one file made about an authority nobody could consult offline.
The value is retired rather than reused. Files exist that carry a 2, so
making that byte mean something else is exactly the change that cannot be taken
back. A reader treats it as 1, because every file that claimed 2 also met
1.
A file therefore says, of its own accord, how much of itself is self-describing — "five streams: three well-formed, two undeclared" — and a reader learns that from the summary rather than from a person.
nominal_rate_millihz is what the stream was configured to produce, in
millihertz so that 200 Hz and 59.94 Hz are both exact. It is not what the stream
achieved; §13 carries that. Both are needed to resample onto a common grid
without inventing data, and a consumer holding only one of them cannot tell a
slow device from a lossy link.
origin describes the data and constrains nothing. raw from an
instrument, derived from a computation, ephemeral for display. A sample lost
from a raw stream is gone; one lost from a derived stream can be recomputed if
its input survives. Nothing in this format derives a policy from the value — it
records what the data is.
flags bit 0, payload_entropy_coded, declares that payloads are already
compressed — a JPEG frame, an H.264 packet, an Opus frame. A writer uses it to
choose a codec. The container names no schema in any compression decision;
the property is declared per stream by whoever configured it, because a
container that special-cased ca3.canonical.v1.VideoFrame would have learned a
vocabulary it must not know.
5.8 Tensor forms declare their shape
A tensor form is the array a schema's numbers project onto, declared as a
TensorForm message on the tensor_form option. §5.2 clause 6 requires it to
name the fields it takes, their order and the resulting shape, and this is what
those three mean.
The case it exists for is use case 4. A body is 28 joints of 8 numbers each at
60 Hz, an EEG chunk is 32 channels of 250 samples, and a consumer training a
model on either needs one array with a known shape, a known element type and a
known axis order. Protobuf fields say what each quantity means and say nothing
about the array — so every consumer picks an order, and two consumers pick
differently. [channel, sample] and [sample, channel] are the same numbers
and not the same tensor, and nothing downstream detects the disagreement.
The test for whether a schema should declare one
A form declares what the schema fixes. A schema that fixes nothing about its array has no form, and saying so is a declaration rather than an omission.
A runtime extent is fine: eeg.tensor.v1 is [channel, sample] with both
lengths decided by the stream, and it is worth declaring because the element
type, the unit and the axis order are fixed by the schema and are exactly what a
consumer would otherwise guess. A runtime element type is not: a form that
cannot say what its elements are cannot say how long its own block is, and a
consumer would have to read a field before it could use the form at all.
Two further clauses, and they are the ones that produce the interesting answers:
- A form may take some fields and leave others.
gaze.tensor.v1takes the three eyes and leavesscreen_point, because a screen point is in a different space and belongs to no eye. A field outside a form is read as a field. - A form may take a field only where the schema can tell a measurement from
its absence — either the field is always reported, or a validity component
covers it, or the value a producer leaves when it has nothing is not a value
the quantity can take. This is §5.6 applied to a shape rather than to a field,
and it is why
PointerSampledeclares no form:(0, 0)is where a pointer sits more often than anywhere else on a screen, and nothing in that message says whether a producer meant it.
The two sources
| Source | block | component.from | Fixes |
|---|---|---|---|
| Block | the bytes field that is the array | empty | an exact payload length |
| Projection | empty | the field each component is taken from | an assembly order |
A block form is bytes a producer packed, and a reader can refuse a block that is not 896 bytes without decoding it. A projection form is an array a consumer assembles out of ordinary fields, and there is no length to refuse.
Row-major, always
The layout is fixed for every form rather than declared. A form with two possible layouts is a form a consumer transposes silently: the numbers stay finite and plausible and the joints move. A producer holding column-major data transposes before packing.
An axis states its length in exactly one of four ways
| Length | Example | |
|---|---|---|
length | fixed by the schema | body.tensor.v1's 28 joints |
from | the number of fields listed, one entry each | gaze.tensor.v1's three eyes |
from_repeated | the length of that repeated field | eeg.tensor.v1's channels |
length_from | the value of that integer field | eeg.tensor.v1's samples |
length_from resolves against the message the form is declared on and never
against a row type an outer axis arrived at. The difference is load-bearing for
EEG: an invalid channel carries an empty microvolts, so an extent taken from
that field would make one row shorter than the rest and the array ragged. The
declared count is how wide every row is; the validity channel is what says which
rows hold nothing.
What a validator MUST check
Each of these is a disagreement between two statements the descriptor already carries, which is what makes them checkable where a comment was not:
- The form declares an identifier, and it ends in a version segment. A form is never edited — a different shape, component order or element type is a new version, and both stay valid at once.
- The form declares an element type, and a block form declares a byte order.
- Every axis has a name, no two axes share one, and every axis states its length in exactly one of the four ways above.
- A block form's
blocknames abytesfield of the message, and that field's declaredcardinalityis the first axis's length. - The number of components is the last axis's length — or, where the last axis takes its length from a field, there is exactly one component and the field it names is repeated.
- An axis
labelled_byan enum names one the descriptor carries, whose values are exactly the indices0 … n-1. The enum value is the index, which is whatBodyJointnumbers itself from zero for; an enum that had drifted into protobuf's unspecified-at-zero convention would still have the right number of names and would label every row off by one. - Every component's declared unit is valid UCUM.
- Every projection component's unit is the unit of the field it is taken
from. This is the one check in the mechanism that catches a mistake rather
than an omission: a component claiming metres taken from a field annotated
millimetres is two statements that cannot both be true, and nothing else in a
descriptor can see it. The unit is inherited down the path, so a component
taking
acceleration.xis checked against them/s2onacceleration. - Every projection component's field has the form's element type, or is a
bool. Aboolcontributes 1 or 0, which is how a validity channel gets into an array of numbers. Nothing else is widened.
A component with no unit is reported as unchecked, never folded into a pass:
a descriptor cannot tell a measurement from a converter count, and
PpgChannel.counts is the case where declaring one would be false.
What a validator cannot check
That the producer packed the components in the declared order. A transposed axis produces finite, plausible numbers and nothing in a descriptor detects it. The reference implementation holds its one packer to a byte-exact expectation for exactly this reason; a schema with several producers has no such guard.
That the numbers behind an invalid entry are unreadable. Where a form has a
validity component, or a validity channel travels beside it, that channel is
what a consumer tests. body.tensor.v1 writes NaN into every numeric component
of an untracked row, so a consumer that ignores the validity component gets
something it cannot mistake for a measurement — but that is a property of the
one packer, not of the form. A projection form is assembled from fields any
producer filled, and its invalid entries hold whatever was left there. §15.2
carries both.
5.9 A tensor block may declare a boundary
A stream MAY declare that its tensor blocks begin at a file offset which is a
multiple of a stated power of two. The declaration is two fields on the
Stream record, and both default to zero:
| Field | Meaning |
|---|---|
block_alignment | The boundary, a power of two up to 4096, or zero for no claim |
block_offset | Bytes from the start of a message payload to the first byte of the tensor block. MUST be zero when block_alignment is zero |
The guarantee, stated exactly. When a stream declares block_alignment A and
block_offset K, then for every message on that stream, the file offset
of the payload's byte K is a multiple of A.
The base is the start of the file, byte zero of the .ca3 — not the start
of a chunk, a section, or a record. Saying which base is not pedantry: a
guarantee relative to a chunk is worthless to a consumer mapping the file,
because the chunk's own offset is then unconstrained and the two congruences do
not compose.
block_offset is stated rather than discovered, because this container does
not parse payloads. A block is a protobuf bytes field, and its distance
from the start of the message is a fact about the schema. The producer — the
layer that serialised the message — knows it; the container must not learn it,
because a container that parsed payloads would need every schema it ever
carried. It is one number per stream rather than one per message because a
§5.8 block form fixes the block's length, which fixes its tag and length
prefix, which fixes the distance for every message on that stream.
It is optional, and declining it is not a degraded mode
Zero is the ordinary case. A writer that declares nothing is fully conformant, and so is a reader that never looks. Three reasons this is a declaration rather than a requirement, and each is a case where the cost is real:
- Padding costs bytes. The mechanism below spends a chunk header and up to one alignment of filler per message. On a dense block stream that is a few per cent; on a stream of small messages it can exceed the data several times over. A recording made to a constrained device declines it.
- Alignment constrains the writer. Honouring it means flushing a chunk per
message, which is buffering and latency a capture path may not be able to
afford.
docs/decisions/0003-cost-that-can-be-paid-after-recording.mdis why this is a converter's job first and a writer's job second. - Files exist that predate this section. They carry no declaration, decode as zero, and are exactly as valid as they were. Absent and zero are the same statement — no claim — so nothing has to be migrated and nothing is retroactively non-conformant.
A reader MUST learn the guarantee from the declaration and MUST NOT infer it from the bytes. A block that happens to land on a boundary in one chunk says nothing about the next, and a reader that probed one and assumed the rest would be right until it was quietly wrong.
The boundary is a parameter, and the choice is not obvious
The stream states the number rather than this document fixing one, and that is a decision with reasons on both sides rather than a deferral.
| Boundary | What it buys | What it costs |
|---|---|---|
| the element width — 2, 4, 8 | The minimum for a correctly-typed load. Below this some architectures fault and the rest are slow | Almost nothing |
| 64 | A cache line, and the widest SIMD load in common use | Up to 64 bytes per message |
| 4096 | A page: the granularity a memory map works in, and the alignment a pinned DMA or RDMA buffer already has | Up to 4096 bytes per message |
Page alignment is the one that closes an end-to-end path. A capture chain in which a network card writes camera frames directly into pinned, page-aligned host buffers can carry a frame from the wire to a tensor without a copy — but only if every stage agrees on the boundary. A container guaranteeing 64 where the source produced 4096 does not corrupt anything; it reintroduces a copy at the container boundary because two numbers disagree. A format that fixed 64 would foreclose that path for every file, permanently, to save padding on streams that could have declined the guarantee instead.
And the cost of the larger boundary is not uniform — it is inversely proportional to the payload. A 4096-byte boundary on a 4 MB camera frame is at most 0.1 %; on a 30-byte inertial sample it is over a hundredfold. That spread is the reason this is declared per stream rather than per file: one recording holding dense frames beside sparse events wants the guarantee on the first and emphatically not on the second, and a whole-file switch cannot say so.
So: no default is right for every stream, which is what makes the parameter the answer rather than a number in this document. A producer or converter picks it from what the consumer will do with the file.
What a writer does
Nothing in the message encoding can carry padding. §6.2 puts messages back to back inside the records blob with no framing of their own, so there is no field between two messages that a pad could hide in, and §1.4 rule 2 forbids inflating an existing varint to make room — a varint MUST be the shortest encoding of its value. So padding goes between chunks, and the unit of alignment is therefore a chunk.
A writer honouring the declaration therefore:
- Emits one message per chunk on that stream. Only the first payload in a
chunk can be moved by a record placed in front of it, so every payload must
be a first payload (
crates/container/src/writer.rs:388). - Computes where the block would land — the chunk record's offset, plus
§3's nine framing bytes, plus the payload's offset within the chunk content,
plus
block_offset. The payload offset is asked of the decoder rather than predicted (crates/container/src/chunk.rs:628,payload_offset), so it cannot drift from §6.2 when a field-mask bit is assigned. - Inserts a
Paddingrecord (0x4A, §13.4) of whatever length puts that offset on the boundary (crates/container/src/writer.rs:517,pad_for_alignment). APaddingrecord cannot be shorter than its own nine-byte frame, so a shift shorter than nine is raised by whole multiples of the alignment until it is expressible (crates/container/src/align.rs:115,padding_before).
This is an additive change and not a version bump. Padding sits in
§3.1's 0x40-0x7F skippable range, so a reader that has never heard of this
section steps over it and reads the same messages; the two Stream fields are
appended, so §14's mechanism 2 applies and a reader that stops after metadata
reads what it always read. No required_features bit is claimed, because
no reader needs to understand anything new in order to be correct — only to
benefit.
sed -n '/for opcode in \[Opcode::Statistics/,/}/p' crates/container/src/opcode.rs |
grep -oE 'Opcode::Padding'
sed -n '/fn pad_for_alignment/,/^ }/p' crates/container/src/writer.rs |
grep -oE 'Opcode::[A-Za-z]+' | tr '\n' ' '
The additive property is that the two fields are written after every field a reader from before this section knows about:
awk '/^impl Stream \{/,/Decode from a record/' crates/container/src/records.rs |
grep -oE 'self[.](https://github.com/extendedresearch/ca3/blob/main/spec/metadata|block_alignment|block_offset)' | tr '\n' ' '
And the largest boundary a stream may name is a page, which is what keeps the mapping and DMA cases expressible:
grep -oE 'MAX_ALIGNMENT: u32 = [0-9]+' crates/container/src/align.rs | grep -oE '[0-9]+$'
A reader meeting a value that is not zero and not a power of two within that
bound MUST refuse the stream, naming it
(crates/container/src/align.rs:83, is_valid_alignment);
crates/container/src/records.rs:820 is the field and
crates/container/src/records.rs:937 the predicate a reader asks.
A file offset is not an address, and only the address matters
This is the half of the mechanism that lives outside the format, and it is
stated here because a reader that implements only the format half gets nothing
for it. A guarantee that a block begins at file offset k * 64 buys a
consumer exactly nothing on its own: the reader still copies those bytes into a
buffer whose allocation is aligned to 1, at whatever position the chunk framing
happens to put them.
A conforming reader that wants the benefit allocates so that
address = file_offset (mod block_alignment) holds for every byte it read,
and then a byte the writer placed on the boundary is read on the boundary.
crates/container/src/align.rs:148 (AlignedBytes) is that buffer and
crates/container/src/reader.rs:751 (read_chunk_aligned) is the read.
Neither half is useful alone. An aligned buffer over an unaligned file gives an aligned buffer and an unaligned tensor; an aligned file read into an ordinary buffer gives an aligned file and an unaligned tensor.
A consumer SHOULD test the address it actually got
(crates/container/src/align.rs:231, is_aligned) rather than trusting the
declaration, and copy when the test fails. The declaration says what the writer
promised; the address is what the tensor is built on.
What this does not do
It does not make several messages one tensor. §6.2 separates consecutive
payloads with the next message's len and timestamp deltas, so N messages
are N runs of bytes and never one. A consumer batching N blocks into one
[N, ...] array copies them together, and this section removes the alignment
problem from each block rather than the batching copy from the set. A
column-major chunk is what would change that, and §15 reserves 0x0E for it
against a trigger this does not fire.
It does not remove the read. One copy from the operating system into the reader's buffer remains, and every copy after it is what goes. Removing the read as well means mapping the file, which is a reader's choice this document does not constrain and this repository does not implement.
6. Chunk (0x05) and the message encoding
A chunk belongs to exactly one stream. This is the format's keystone.
Single-stream chunks make a stream's bytes a set of owned extents, which is what makes a selective-stream read cheap: consult the index, fetch that stream's chunks, decompress nothing else. In a session where video is ninety-five percent of the bytes, reading gaze and inertial data touches roughly two percent of the file. A container with heterogeneous chunks must decompress the blocks the wanted messages sit inside, which is most of the file, because compressed frames are not randomly addressable.
That access pattern is use case 4 — training on a subset of streams, repeatedly, across a corpus. It is also every analysis that wants one modality.
What it costs, stated: structural overhead per chunk falls on every stream regardless of rate, so a low-rate stream pays a large fraction. §15 carries the measurement and the condition that would reverse the decision.
6.1 The chunk header
offset size field
0 4 stream_ref u32
4 1 codec u8 0 none, 1 zstd
5 2 field_mask u16 §6.3
7 2 flags u16 bit 0: sorted. Others reserved, zero
9 4 message_count u32
13 8 min_stamp_ns u64 the delta base
21 8 max_stamp_ns u64
29 8 record_ts_base_ns u64
37 8 wall_base_ns u64
45 8 source_base_ns u64
53 4 seq_first u32 0 = absent; §6.4
57 8 uncompressed_bytes u64
65 4 records_crc32c u32 CRC-32C over the blob as stored
69 8 records_length u64
---------
77 fixed header
The framing prefix adds 9, so the records blob begins at
chunk_offset + 9 + 77 = chunk_offset + 86.
How the bases are computed. Each is a minimum over the chunk's messages, and zero when no message carries the field:
min_stamp_ns— the minimummonotonic_timestamp_ns, which keeps everystamp_deltanon-negative even when a caller supplies messages out of arrival order.record_ts_base_ns— the minimumrecord_ts_ns, for the same reason.source_base_ns— the minimumsource_timestamp_ns.wall_base_ns— the minimum ofwall_clock_ns − stamp_delta, since bit 1 stores a skew againstwall_base_ns + stamp_deltarather than a value.
One rule for all four rather than a different rule for the last two, which was an inconsistency and not a design. A reader is unaffected either way — every base is stated in the header and the per-message deltas of bits 1 and 2 are zigzag-coded, so the choice changes a varint's width and never a decoded value — but two writers following different rules produce different bytes for identical input, and §16's corpus is a byte comparison.
flags bit 0, sorted, is set when the writer emitted this chunk's
messages in non-decreasing monotonic_timestamp_ns order.
It is an optimisation and never a correctness condition. §9's within-chunk walk stops at the first stamp past the query when the bit is set, because no later record can carry an earlier stamp. A reader that ignores the bit and walks every record returns the same message; a reader that believes a wrongly set bit returns a wrong one. A writer therefore sets it from the order it actually emitted, never from an assumption about its input.
codec. 0 = none, 1 = zstd, and no other value is assigned. An
unrecognised codec is unreadable rather than skippable — you cannot decompress
what you cannot decompress — so every codec addition claims a
required_features bit and refusal happens at open. A reader meeting an
unassigned value MUST reject the file, name the value, and MUST NOT treat
the blob as uncompressed.
Chunk cannot grow by appending fields, because its last field is followed
by an unlength-prefixed blob. A chunk-header change takes a new opcode. This is
why a columnar chunk is a separate record rather than a flag here.
6.2 Message records
Messages live in the blob back to back, with no opcode and no nine-byte frame. Per-record framing is paid on structural records, of which a session has thousands, and never per message, of which it has billions.
len : varint bytes after this field, for this message
stamp_delta : varint monotonic_timestamp_ns − chunk.min_stamp_ns
<one field per set bit of field_mask, ascending bit order>
payload : the remaining bytes of len
Payload bytes are copied through untouched. This is not a re-serialisation of a
transport envelope: the constant facts live once on the Stream record, and the
timestamps become deltas against the chunk's bases.
6.3 The field mask
| Bit | Field | Encoding | Absent |
|---|---|---|---|
| 0 | record_ts_ns | varint delta from record_ts_base_ns | not absent when the bit is set |
| 1 | wall_clock_ns | varint v; v = 0 absent; else wall_base_ns + stamp_delta + zigzag(v − 1) | v = 0 |
| 2 | source_timestamp_ns | varint v; v = 0 absent; else source_base_ns + zigzag(v − 1) | v = 0 |
| 3 | sequence | varint v; v = 0 absent; else wrap(seq_first + i + zigzag(v − 1)) | v = 0 |
| 4–7 | reserved, must-understand | — | a reader MUST reject a chunk with any set |
| 8–15 | reserved, skippable | exactly one varint each, ascending bit order | — |
field_mask is the union over the chunk's messages.
The wall clock is stored as a skew, not as a value. Wall and monotonic clocks
advance together, so the difference between them barely moves; encoding
wall − (wall_base + stamp_delta) puts a one- or two-byte varint where an
absolute timestamp would cost eight.
Bits 8–15 are defined in advance to be exactly one varint each, in ascending bit order, which is what lets a reader that has never heard of a future field skip it and still land on the payload boundary. Bits 4–7 are held for a field whose presence would change what a message means, and a reader that meets one stops.
6.4 Sequences
i is the message's zero-based index in the chunk.
wrap(x) = ((x − 1) mod (2³² − 1)) + 1: sequences run 1..=u32::MAX and wrap
back to 1, skipping 0 so that absent and zero stay distinguishable.
Getting that modulus wrong by one turns a single wrap into a reported loss of one sample, once per 4.29 billion messages — about every fifty days on a one-kilohertz stream, which is long enough that nobody would connect the report to the cause.
When bit 3 is clear, seq_first decides for the whole chunk. Zero means
every message's sequence is absent; non-zero means message i has
wrap(seq_first + i). A writer clears the bit when the run is contiguous, which
is the ordinary case and costs nothing, and equally when nothing carries a
sequence at all, which also costs nothing. Setting it means at least one message
deviated, and then every message pays at least one byte.
If the first message's sequence is absent while a later one carries it,
seq_first is 0 and bit 3 MUST be set. There is no base to count from.
A reader that derives wrap(0 + i) for a chunk in which no message carried a
sequence invents a counter no producer stamped — and because loss detection is
built on sequences, the fabricated run reads as a clean one and the reader
reports no loss with total confidence.
6.5 ColumnarChunk (0x0E)
The same messages as §6.2, laid out so that sample n is an address rather than a walk. A converter produces one; no recorder is required to, and §6.2 remains the ordinary chunk.
offset size field
0 4 stream_ref u32
4 1 codec u8 0 none, 1 zstd
5 2 field_mask u16 §6.3's bits, each a column here
7 2 flags u16 bit 0: sorted
9 4 message_count u32
13 8 min_stamp_ns u64 the stamp column's base
21 8 max_stamp_ns u64
29 8 record_ts_base_ns u64
37 8 wall_base_ns u64
45 8 source_base_ns u64
53 4 seq_first u32 §6.4's meaning exactly
57 4 block_bytes u32 bytes of payload per message
61 8 columns_length u64 the column region, as stored
69 8 blocks_length u64 the block region, as stored
77 4 payload_crc32c u32 over columns then blocks, as stored
---------
81 fixed header, then the column region, then the block region
Why the fields become columns. §6.2 interleaves: each message carries its
len, its stamp_delta, one value per set field_mask bit, and then its
payload, back to back with the next. Reaching sample n means parsing the n
before it. Here each field is message_count values of one width, one column
after another, and the payloads follow as a single region of
message_count × block_bytes.
Column order is the stamp, then ascending bit order — the same order §6.3 gives a message's fields, so a reader that knows one knows the other. Widths:
| Column | Width | Value |
|---|---|---|
| stamp | 8 | monotonic_timestamp_ns − min_stamp_ns |
bit 0 record_ts_ns | 8 | delta from record_ts_base_ns |
bit 1 wall_clock_ns | 8 | §6.3's skew; 0 absent |
bit 2 source_timestamp_ns | 8 | delta from source_base_ns; 0 absent |
bit 3 sequence | 4 | §6.4's value; 0 absent |
The widths are fixed where §6.3's are varints, and that is the trade this
record makes. A varint cannot be indexed — its length depends on its value —
so a column has to be strided to be addressed at all. Density is given up for
the seek, which is why this is a converter's output and not a writer's
(docs/decisions/0003-cost-that-can-be-paid-after-recording.md). The
v = 0 absent convention is kept exactly, so a conversion between the two
layouts loses nothing in either direction.
Bits 4–7 are must-understand here as in §6.3, and a reader MUST reject a chunk setting one. Bits 8–15 have no defined width in this layout and a writer MUST NOT set one; a reader MUST reject a chunk that does, rather than guessing a stride.
Sample i's block begins at block_bytes × i of the block region. That
multiplication is the whole reason the record exists.
A reader MUST reject a chunk whose columns_length or blocks_length
disagrees with the widths above and message_count. Both are derivable from
fields the header already carries, so a file that states them differently is
stating one fact twice and disagreeing with itself — and the consequence is not
cosmetic. A short columns_length moves where the block region begins, so
every block_at in the chunk then addresses somewhere else, and the bytes
found there decode into a plausible tensor rather than into an error.
A rectangular selection is arithmetic too. §5.8 fixes a block form's element type and a row-major layout, so component k sits at the same offset inside every block, and the byte ranges for samples i..j × components k..l are computable without reading anything. They are ranges and not a range — a strided selection is not contiguous in the file, and returning one span would hand a caller bytes it did not ask for, interleaved with the ones it did.
A stream whose payloads are not one fixed length has no home here.
block_bytes is one number for the chunk, so a writer MUST refuse rather
than pad. §5.8's block forms fix an exact payload length and are what this is
for; projection forms and undescribed payloads stay in §6.2 chunks.
Compression and addressing are exclusive. A codec other than zero means
the payload region must be decompressed before any offset in it means
anything — the same trade a compressed chunk makes in any format. A file
prepared for random access uses codec 0.
A file containing one ColumnarChunk MUST set required_features bit 2
(§3.3), so a reader that has not implemented this section refuses the file at
open by name rather than meeting a must-understand opcode three gigabytes in.
This record cannot grow by appending fields, for §6.1's reason: its header is followed by two unlength-prefixed regions. A layout change takes another opcode.
7. The chunk index and the ordering invariant
7.1 ChunkIndex (0x08)
One packed, fixed-stride record. Fixed stride is what makes the index binary-searchable directly on a memory map with no parsing.
entry_size : u32 // readers stride by this value
entry_count : u32
entries : entry_count × entry_size bytes
A reader MUST use entry_size as the stride and MUST NOT assume a constant.
A later version appends fields and raises the value; the leading bytes keep
their meaning. A reader MUST reject an entry_size smaller than this
version's entry.
offset size field
0 8 chunk_offset absolute offset of the opcode byte
8 8 chunk_length 9 + content_length
16 8 min_stamp_ns
24 8 max_stamp_ns
32 8 prefix_max_stamp_ns running max of max_stamp_ns over this
stream's entries, in table order
40 4 message_count
44 4 stream_ref
48 4 seq_first 0 = absent
52 4 seq_last 0 = absent
---------
56 entry
Entries are sorted by (stream_ref, min_stamp_ns) — by stamp, not by file
position, which is what keeps the binary search exact when messages arrived out
of order.
7.2 The ordering invariant
The obvious seek — binary search for the last chunk whose min_stamp ≤ T, and
take its last message — is wrong the moment two chunks of one stream overlap in
time, because an earlier chunk can then hold a later message.
Invariant. For each stream, taken in min_stamp_ns order — which §7.1 is
the order the index table is in — the intervals
[min_stamp_ns, max_stamp_ns] MUST NOT overlap:
min_stamp_ns(n+1) > max_stamp_ns(n).
In min_stamp_ns order and not in write order, because §7.1 sorts the
table and sorting is what repairs a stream whose chunks were merely written
out of order while their spans stayed disjoint. What sorting cannot repair, and
what this invariant is therefore about, is one chunk's span reaching past the
start of the next — an earlier entry in the table holding a message later than
anything in the entry a binary search lands on.
Enforcement. A writer MUST achieve it where it can and MUST declare
failure where it cannot, by setting the chunks_overlap bit on that stream's
directory entry. The bit is derived from the spans the file actually
carries, never asserted independently of them: a reader that believed a bit
disagreeing with the bytes would take the wrong one of §9.1's two paths, and
the wrong one returns a plausible answer rather than an error.
A writer MUST NOT drop a message to preserve the invariant. Every lost sample being detectable is the point of §6.4, and a container that silently discards a late arrival is that detection failing at the last step.
How a writer achieves it. Messages buffer per stream until the chunk flushes.
Within an open chunk, arrival order does not matter — min_stamp_ns and
max_stamp_ns are computed over its contents at flush. The invariant can only
break when a message arrives after its stream's previous chunk was sealed,
carrying a stamp at or below that chunk's maximum. The writer's reorder window
is therefore exactly the chunk span.
What a writer does with a late arrival. It appends the message to the open
chunk, sets chunks_overlap for that stream, and counts it. The data is kept,
the query degrades, and the degradation is on the record.
7.3 StreamDirectory (0x09)
One entry per stream, so a reader jumps to a stream's contiguous sub-array of index entries without scanning.
entry_size : u32 // the FIXED PREFIX below, not the whole entry
entry_count : u32
entries : {
stream_ref : u32
flags : u32 // bit 0: chunks_overlap
first_index : u32 // index of this stream's first ChunkIndex entry
index_count : u32
message_count : u64
chunk_count : u64
first_stamp_ns : u64 // in this stream's own domain
last_stamp_ns : u64 // in this stream's own domain
dropped_by_queue: u64 // discarded after arrival, by this recorder
never_arrived : u64 // absent from the sequence, never seen
reordered : u64 // arrived after a later-stamped message
sync_quality : u8
--- fixed prefix ends: entry_size bytes ---
sync_domain : string
}
entry_size covers the fixed prefix and stops before sync_domain. A
clock domain's name has no fixed width, so this table is not stride-addressable
the way §7.1's is, and saying otherwise would tell a reader to seek to a byte
that is not an entry boundary. What entry_size still buys is the growth rule:
a later version appends fixed fields and raises the value, and a reader steps
over the ones it does not know before reading the string. A reader MUST
reject an entry_size smaller than this version's prefix.
dropped_by_queue and never_arrived are different numbers and a single
"lost" figure could not be acted on. One says a consumer could not keep up;
the other says the data never reached this machine. The first is fixed by
buffering, the second by the link.
These are the recorder's own observations, and they are stronger than anything a reader can infer. A reader can find gaps in a sequence; it cannot know whether the recorder saw a message and dropped it. That is a fact about an instant, and §0.3's G3 reasoning applies — capture it or lose it.
This record is the corpus-index primitive. Filtering a thousand sessions to those where a stream had under five percent loss is a summary read per file, not an open-and-scan. A reader MUST be able to satisfy it from the summary alone.
7.4 A sequence gap is not always a loss
A gap in one stream's recorded sequences is evidence of a drop only if that stream is the only stream on its topic in this file.
Where a topic carries more than one stream, each counts independently from 1, so
read as one series they interleave into apparent duplicates and apparent gaps. A
reader MUST NOT report a per-stream gap as a loss without first counting the
streams on that topic, which it can do from the Stream records alone.
7.5 MessageOffsets (0x49) — skippable
Where each message of one chunk begins. One record per indexed chunk.
chunk_offset : u64 absolute offset of the chunk's opcode byte
entry_size : u32 readers stride by this value
entry_count : u32
entries : entry_count × entry_size bytes
offset size field
0 4 message_offset u32 bytes from the start of the records blob
What it buys, with the measurement. §6.2 gives a message no address, so
reaching sample n means walking the n before it — and reaching it from a
file means fetching the whole chunk, because there is no smaller range to ask
for. measurement/baselines measured ten thousand shuffled reads over a
hundred thousand samples in chunks of a thousand: 8,612 MiB fetched,
against 8.5 MiB for the same reads over §6.5's layout. With this table a reader
fetches one message, and a ~900 KiB chunk read becomes a ~900 byte message
read.
Offsets are blob-relative, not file-relative. The alternative ties the table to the chunk's position, so a converter that moved a chunk would rewrite every entry rather than one header field.
u32, because §1.2 caps a bytes field at u32::MAX and a chunk's records
blob is one. Eight bytes an entry would double the table to describe a range
that cannot occur.
The last message's end is not stored. It is the blob's end, which the chunk header already gives. Storing it would be a second statement of one number, and two copies of a fact are how they come to disagree.
Entries MUST be strictly ascending, and a reader MUST reject a table that is not. A backwards entry sends a seek into the middle of an earlier message, where the bytes decode into something plausible rather than failing — the silent-wrong-answer shape §3.1 partitions the opcode space to prevent.
Skippable, and derivable, and those are the same fact. A reader that ignores this record returns the same messages, having walked for them; anything holding the chunk can rebuild the table by walking once. A file that lost it lost speed and no information, which is what lets it live in the summary rather than beside the chunk it indexes.
A converter writes it; no recorder is required to. Building the table costs
a walk of every chunk, which is a cost a consumer wants and the capture path
should not pay — docs/decisions/0003-cost-that-can-be-paid-after-recording.md.
8. Checkpoint (0x07) and recovery
A format whose index lives only in the tail loses everything when the tail never arrives. For a battery-powered recorder that is the expected ending rather than the exception.
A checkpoint inlines every index entry produced since the previous checkpoint and back-points to it.
8.1 Layout
checkpoint_seq : u64
flags : u32 // bit 0: restates every live definition
reserved : u32 // MUST be zero; a reader ignores it
record_ts_ns : u64 // recording host monotonic clock at emit
wall_ns : u64 // recording host wall clock at emit
schemas : array<bytes> // each element one Schema record's content
streams : array<bytes> // each element one Stream record's content
index_entries : { entry_size: u32, entry_count: u32, packed entries }
clock_block : bytes // ClockBlock-content as of now, or empty
--- fixed 36-byte trailer ---
sentinel : 8 bytes "CA3POINT"
self_offset : u64 absolute offset of THIS record's opcode byte
prev_offset : u64 absolute offset of the previous Checkpoint; 0 first
checkpoint_seq : u64 copy, for validation
content_crc : u32 CRC-32C over the content preceding the sentinel
Index entries exist in exactly two places — checkpoints, incrementally, and the summary, complete and sorted. One mechanism, not two. The cost is roughly double index storage.
An ordinary checkpoint carries only definitions declared since the previous one, and the backward chain reaches the rest. Periodically a checkpoint sets bit 0 and restates every live definition, so a file whose middle is damaged resolves its schemas from a nearby checkpoint rather than only from an unreachable head.
8.2 The self-locating trailer
Walking a backward-linked chain requires finding the last checkpoint, and a truncated file has no footer to point at one. A backward byte scan would otherwise be unsound, because record bodies hold arbitrary compressed bytes that may contain anything.
The trailer makes the scan sound by giving every candidate four independent checks it must pass together:
- The eight-byte sentinel matches.
self_offsetS, read atP + 8, is less thanPand at least 9.- The byte at S is
0x07, and theu64atS + 1satisfiesS + 9 + content_length = P + 36. content_crcverifies over the content bytes.
Check 3 is what makes a false positive implausible: a coincidental sentinel
would have to be preceded, at exactly the distance its own coincidental
self_offset names, by a matching opcode byte and a length that lands back on
the sentinel.
8.3 Recovery
recover(file):
# 1 — sealed?
if last 8 bytes == MAGIC:
footer = read(EOF − FOOTER_FROM_END)
if footer.summary_start != 0 and summary_crc verifies:
return indexed_read_from_summary(footer)
# a sealed file with a damaged summary falls through
# 2 — locate the last checkpoint
if footer readable and footer.last_checkpoint_offset != 0:
enter the chain directly
else:
backward-scan for a sentinel, validating each candidate by §8.2
if no candidate: goto 6
# 3 — walk the chain backward, merging definitions and entries;
# a failed content_crc breaks the chain -> 6.
# The newest non-empty clock_block wins.
# 4 — forward seek-skip from the last checkpoint to EOF,
# rebuilding an entry for each complete Chunk found
# 5 — rebuild the derived columns
sort entries by (stream_ref, min_stamp_ns)
recompute prefix_max_stamp_ns per stream
recompute the StreamDirectory, including chunks_overlap
# 6 — last resort: forward seek-skip of the whole record spine
Step 5 is not optional, and it is the one place in this format where an
implementation bug produces plausible wrong output rather than an error.
prefix_max_stamp_ns is derived over the sorted table, and a checkpoint
inlines entries in flush order where no sorted table exists. A reader that
recovers entries and forgets to recompute the column returns silently wrong
answers for any stream with chunks_overlap set. §16 requires a conformance
fixture with deliberately disordered chunks.
8.4 What a truncated file loses
| Recoverable | Not recoverable |
|---|---|
| Every chunk whose framed bytes are complete | Messages still in open chunk buffers when the writer died — bounded by the chunk span, not by the checkpoint interval |
| Every schema and stream the chain reaches | A chunk partially written at EOF |
| The clock block as of the last checkpoint | The summary's derived records — recomputable by a full pass, but not present |
A reader that recovers a truncated file MUST report it as truncated and MUST NOT present the recovered set as complete.
A reader MAY run this against a file still being written.
8.5 Durability is not this format's to enforce
Bytes handed to the operating system are not bytes on the medium. Two writer policies give two different bounds, and a writer MUST state which it implements:
- Synchronise after every chunk and checkpoint — loss is bounded by what sits in open chunk buffers.
- Synchronise only at checkpoints — loss additionally includes every chunk written since the last one.
The cost of either on any target's storage is unmeasured. Until it is, both are knobs and neither is a default.
9. Queries
Two primitives, and they are not variations of one another.
9.1 State at an instant
What was every stream's latest value at time T. The replay scrub.
state_at(T):
for each stream s:
# 1 — move T into s's own domain.
# The INDEX is never mapped; the QUERY is.
if s's domain == the file's domain:
T_s = T ; basis = "exact"
else if no usable mapping for s's domain:
T_s = T ; basis = "unmapped" ; report s degraded
else:
T_s = inverse_map(T, m) ; basis = m.quality
# 2 — binary search this stream's sub-array for the last entry
# with min_stamp_ns <= T_s
# 3 — if chunks_overlap is clear, that chunk holds the answer
# 4 — if it is set, scan back while prefix_max_stamp_ns exceeds the
# best stamp found, and no further
Why the index key is raw domain-local time. Indexing on a mapped estimate
would mean every improvement to the mapping invalidates the index and forces a
file rewrite. Indexing on each stream's own raw timestamps means a better
estimate changes only the transform applied at step 1. A re-estimation changes
map(), never a byte of the file.
This is the mechanism behind G3, and it is a structural constraint rather than a preference.
Within a chunk, the walk reads each record's len and stamp_delta and
steps over the rest, decoding exactly one message. Ties resolve to the later
record — the question is the last message at or before T_s, and a chunk may
carry two at one stamp. On a chunk with sorted set the walk stops at the first
stamp past T_s.
Every emitted result carries its basis. A reader MUST NOT return a time derived through a mapping without saying so.
9.2 A window
Every message between T1 and T2, for these streams. The training and
analysis primitive.
window(T1, T2, streams):
for each requested stream s:
map T1 and T2 into s's domain as in §9.1
select index entries where max_stamp_ns >= T1_s and min_stamp_ns <= T2_s
for each, in min_stamp_ns order:
fetch, decompress, and emit messages whose stamp is in range
Three properties a reader MUST provide, because a training loader depends on all of them:
- Only the requested streams are fetched. No chunk belonging to an unrequested stream is read or decompressed. This is the keystone paying for itself.
- Results are emitted in
monotonic_timestamp_nsorder within a stream, including whenchunks_overlapis set — which then requires a merge across the overlapping entries rather than a concatenation. - The same file and the same window produce the same sequence, every time. Reproducible training depends on it.
A window is not a resample. It returns the messages that exist, with their
own timestamps and their own gaps. Placing them on a regular grid is the
consumer's operation and needs both nominal_rate_millihz and the observed
timing in §7.3 to be done without inventing data.
10. Time
The section G2 rests on. If one part of this document is read closely, this is it, because timing is where multi-instrument studies go wrong and it goes wrong quietly.
10.1 What this format can and cannot do
It cannot make alignment accurate. Accuracy is produced at capture time by hardware and procedure. If a device exposes no probe and accepts no marker, one-way estimation is the best available and the bound is unbounded — that is the deployment, not a gap in the format.
It can make alignment uniform, stated, and checkable. The arithmetic is normative, so two producers observing the same thing report the same number. Every claim carries its own bound. Nothing that was not measured is presented as though it were.
That is a weaker promise than "accurate" and it is the one that can be kept. It
is also the one that creates pressure: a producer reporting bias unbounded
beside one reporting ± 2 ms is visibly worse at the thing this format exists
for, and there is no estimator choice that hides it.
10.2 The error model
A device's timestamp reaches the reference timeline through four terms, and a CA3 file carries every one of them with its own uncertainty and its own provenance:
t_reference(s) = t_device(s)
+ d_device on-device delay [calibration, ± u_d]
+ o(t) clock offset [fitted, ± u_o]
+ b_transport transport bias [method-bounded, ± u_b]
The third term is the one every format carries and the first is the one none of them do. Published measurement puts an eye-tracker-to-EEG offset at tens of milliseconds from device-internal delay alone, against residual clock jitter measured in hundreds of microseconds. A file that reports the small term precisely and omits the large one has not described its own alignment.
The terms are exclusive, not cumulative. Each is measured by a different route, and two routes can cover the same ground — an alignment event's correspondence already contains the device delay. §10.11 states which applies when, because applying both is how a file produces a plausible number instead of an error.
There is a fifth term this format cannot carry and names anyway: loss inside the instrument. When a driver stops reading, the device's own buffer discards, with no sequence number and no record. Waiting does not prevent that loss; it hides it. Nothing downstream recovers it.
10.3 Clock domains and the reference domain
A clock domain is a (host_id, host_clock_epoch) pair, or a device's
declared source_clock_domain. Never a machine name alone — §4.1.
A file declares one reference domain, and every other domain carries a mapping onto it. Pairwise mappings between every pair are quadratic and mostly absent; a declared reference makes "what is this session's timeline" a fact in the file rather than a convention in the reader.
The reference domain MUST be a domain the file actually carries — ordinarily
the recording host's. A synthetic session timeline starting at zero would itself
be a derived quantity, and §0.3's G3 forbids storing derived time as though it
were measured. Session-relative time is sample_stamp − anchor_monotonic_ns,
computed by a reader.
10.4 ClockBlock (0x0A) — must-understand
Must-understand, because a reader that skipped it would present timestamps from different domains as though they were comparable.
reference_host_id : string
reference_host_clock_epoch : string
domains : array<ClockDomain>
degraded_intervals : array<{ stream_ref: u32, from_ns: u64,
to_ns: u64, reason: string }>
ClockDomain {
domain : string // a source clock domain, or a host_id
kind : u8 // 0 device, 1 peer host, 2 self
method : u8 // §10.5
quality : u8 // 0 unavailable, 1 degraded, 2 ok
offset_ns : i64
skew_ppb : i64 // parts per billion; §1.1
reference_source_ns : u64 // the origin the line is fitted about
residual_spread_ns : u64 // PRECISION
min_rtt_present : u8 // presence precedes the value it governs
min_rtt_ns : u64 // 0 when min_rtt_present is 0
bias_low_ns : u64 // ACCURACY. BIAS_UNBOUNDED = u64::MAX
bias_high_ns : u64 // ACCURACY. §10.5
skew_uncertainty_ppb : u64 // SKEW_NOT_ESTIMATED = u64::MAX
observation_count : u64
observations_stream : u32 // stream_ref of the raw observations, or 0
estimator_version : u32 // which normative estimator produced this
}
reference_source_ns is not optional. Offset and skew describe a line, and
a line needs an origin. Storing the first two without the third makes the
mapping unreproducible, which defeats recording it.
min_rtt_ns carries an explicit presence flag because absent and zero are
different answers. Zero is a trigger correspondence with no path to delay it;
absent is a one-way method that never measured one.
The flag precedes the value, which is the only ordering question in this
record and the one this document had backwards until 2026-08-11. The listing
above is the wire order, so a reader that takes the fields in the order written
lands on bias_low_ns where the writer put it. Reading them the other way
round misaligns by one byte from that point and corrupts every accuracy field
§10 exists to carry — bias_low_ns, bias_high_ns, skew_uncertainty_ppb,
observation_count, observations_stream, estimator_version — while every
field before it still decodes. The record then looks partially good rather than
broken, which is the expensive kind of wrong. crates/container has always
written the flag first; this document is what moved.
10.5 Methods, and the bias-bound rule
The bias bound is two numbers, not one. bias_low_ns and bias_high_ns
are how far below and above the estimate the true offset may lie. A symmetric
bound sets both; an asymmetric one does not, and the asymmetry is not a corner
case — it is the ordinary result of the ordinary method.
| Value | Method | bias_low_ns | bias_high_ns |
|---|---|---|---|
| 0 | receipt_only | BIAS_UNBOUNDED | 0 |
| 1 | one_way_min_filter | BIAS_UNBOUNDED | 0 |
| 2 | round_trip | min_rtt_ns / 2 | min_rtt_ns / 2 |
| 3 | trigger | the anchor's uncertainty | the anchor's uncertainty |
Better wins outright. Where observations of more than one kind exist, the mapping carries the best method available and the bound that method implies.
This table is normative arithmetic, not a convention. It is the difference between a number that describes accuracy and one that only looks like it, and if each producer derived it independently the numbers would stop being comparable — which is the whole of what this section is for.
Why one-way error has a direction. Every observation gives
delta = receipt − source = offset + delay, and delay is never negative. The
minimum observed delta is therefore offset + min_delay, and taking it as the
offset estimate makes the true offset at most the estimate, never more.
A device five milliseconds away with a perfect clock yields an offset estimate of five milliseconds, so this format places its events five milliseconds later than they occurred. One-way estimation never places an event too early. That is a weaker statement than a symmetric bound and a much stronger one than "unknown", and it costs one field.
Why a trigger is not zero. A hardware trigger propagates in nanoseconds, but the instrument detects it on its own sample clock, so the correspondence it establishes is quantised to that instrument's sample period — five milliseconds on a two-hundred-hertz device. Declaring zero would put a false claim on the most trusted row in the table.
A trigger is an alignment anchor (§10.10) reached by a different route, and its bound is that anchor's stated uncertainty. There is one mechanism here, not two.
Why the distinction matters more than it appears. Min-filtering a thousand
one-way arrivals over a link with a steady five-millisecond delay converges on a
residual spread near zero and an offset wrong by five milliseconds, and nothing
in the observations can reveal it. residual_spread_ns is the variance an
estimator removed. bias_low_ns and bias_high_ns are how far the fitted line may sit from the
truth. A consumer shown only the first is shown the number that looks like
accuracy and is not.
BIAS_UNBOUNDED is a measured result, not a missing value. One-way data
cannot bound its own path delay. Reporting it as unbounded is the honest answer,
and there is no better implementation that changes it.
10.6 The mapping, forward and inverse
Both directions are normative, because two readers deriving the inverse differently would place the same query at two different instants.
map_to_reference(t_src, m):
if m.quality == unavailable: return t_src
elapsed = (i64)t_src − (i64)m.reference_source_ns
drift = (i64)((i128)elapsed × m.skew_ppb / 1_000_000_000)
return max(0, (i64)t_src + m.offset_ns + drift)
inverse_map(t_ref, m):
if m.quality == unavailable: return t_ref
s = m.skew_ppb / 1_000_000_000
t_src = (t_ref − m.offset_ns + m.reference_source_ns × s) / (1 + s)
return max(0, round(t_src))
The clamp at zero is not invertible, and that is correct. A source instant whose forward image falls below zero is clamped, and no inverse recovers which of the clamped instants it was. Making that loss visible rather than silent is why the clamp exists.
10.7 Observations travel as an ordinary stream
The raw observations a mapping was fitted from are recorded as a normal CA3
stream, and ClockDomain.observations_stream points at it. They are not a
special record type.
They then get chunking, indexing, compression, the summary, and every query primitive for free, and a reader that wants them uses the tools it already has.
Why they are stored at all, given that nobody re-fits by hand:
- Auditability. A reviewer can check that the stated mapping follows from the stated observations. That is verification, not re-analysis.
- Falsifiability. A mapping with no observations behind it cannot be shown to be wrong, which is a bad property for the number this format rests on.
- Diagnosis. Observations say why alignment degraded — a link that worsened, a device that stopped answering, a clock that stepped.
The mapping in the file is a verifiable cache, not an independent assertion.
It carries estimator_version, and a validator recomputes it from the
observations and MUST report a file inconsistent when the two disagree. The
re-derivation nobody would perform by hand happens mechanically, on every
validation.
A reader that skips the observations stream MUST NOT describe a mapping as verified.
10.8 The estimator is versioned, not replaceable
One estimator per version of this specification. An implementation does not substitute its own.
Version 1 of this document specifies estimator version 1, and a writer
following it sets ClockDomain.estimator_version to 1. The number is stated
here rather than left to each implementation because it is the only thing that
tells a consumer which arithmetic produced a mapping it did not compute, and a
value nobody published cannot be checked against anything.
If each consumer estimated slightly differently, the result would be timestamps that look comparable and are not — a failure that produces plausible results and no error message, which is worse than a crash because a crash tells you.
Improvement happens by specifying a new estimator version. Files written under an earlier one keep valid mappings, stated as such, and a reader that recomputes under a newer estimator MUST report that it did.
10.9 On-device delay
The term no software can measure, carried as a declared calibration with provenance.
DeviceDelay {
device_ref : u32
stream_ref : u32 // 0 = every stream from that device
delay_ns : i64 // positive: the device reports later than reality
uncertainty_ns : u64 // DELAY_UNKNOWN = u64::MAX
method : string // how it was obtained
measured_unix_ns : u64 // when
measured_by : string // who or what
valid_firmware : string // the firmware it was measured under, or ""
valid_config : map // the configuration it holds for
}
A calibration carries the conditions it holds under. A delay measured on firmware 2.1 need not hold on 2.3, and a camera's delay depends on its exposure and frame rate. Without those fields a stale calibration is applied silently and looks exactly like a fresh one.
A reader MUST report a DeviceDelay whose valid_firmware differs from the
Device record's firmware as not applicable, rather than applying it or
ignoring it quietly.
A clock estimator sees data only once it reaches the host. Whatever the instrument did before that — sensor integration, internal buffering, driver latency — is invisible to every method in §10.5.
Measuring it requires the physical world: a signal both the instrument and the recording system observe. So this is a calibration, usually performed once on one unit, possibly by someone else, possibly a year ago — which is exactly why it carries when, by whom, and how, and why a reader must be able to tell it from something measured during this session.
A file with no DeviceDelay for a stream is stating that the delay is
unknown, not that it is zero.
10.10 Alignment events
One physical event, observed by any number of streams.
AlignmentEvent {
event_id : u32
kind : u8 // 0 instrumented, 1 content-derived
method : string // "ttl", "photodiode", "flash", "clap", "xcorr", …
observed_by : string // who or what established the correspondence
observations : array<{
stream_ref : u32
stamp_ns : u64 // in that stream's own domain
uncertainty_ns : u64
}>
}
A flash, a clap, a hardware pulse — anything several instruments physically observe. Because the event starts in the room rather than at a driver, the correspondence it establishes includes device delay, which makes it the only software-recordable measurement reaching §10.2's first term.
Not pairwise. A flash seen by four devices is one event with four observations, not six pairs. The record shape follows the physics, and the combinatorial version would have had six records disagreeing about one instant.
Uncertainty is per observation, because detection is. A photodiode resolves a flash to microseconds; a thirty-hertz camera resolves the same flash to a frame period. One number for the event would throw that distinction away, and it is the distinction that decides which observation to trust.
kind separates instrumented from content-derived. A TTL into a trigger
channel and a cross-correlation of two audio tracks can both establish a
correspondence, and they do not warrant equal trust. A reader MUST be able to
tell them apart without parsing method.
Events are not required. Their absence is required to be visible. A file either carries one with stated uncertainties, or it says the delay is unknown. Both are conformant. This format cannot make anyone flash a light; it can stop a missing measurement from resembling a made one.
10.11 The terms MUST NOT be double-counted
An alignment event measures an offset that already contains the device delay, because it starts at the physical world rather than at a driver.
A reader that applies both a mapping derived from an alignment event and a
separate DeviceDelay for the same stream corrects twice, and the second
correction is the size of the term the whole section exists to capture.
So:
- A
ClockDomainwhosemethodistriggerstates whichAlignmentEventit was fitted from. A reader MUST NOT additionally apply aDeviceDelayto any stream that mapping covers. DeviceDelayapplies only where no alignment event covers the stream.- A file carrying both, with no statement of which was used, is inconsistent and a reader MUST report it rather than choosing.
This is the failure mode that produces a plausible number rather than an error, which is the class this format is built to refuse.
10.12 What a reader returns
Every timestamp a reader emits carries a basis, and a reader MUST NOT return a mapped time without one:
| Basis | Means |
|---|---|
exact | Already in the reference domain; nothing was applied |
ok / degraded | Mapped, with the mapping's quality |
unmapped | No usable mapping; the raw value is returned unchanged |
A reader SHOULD additionally return the total uncertainty, combining the terms of §10.2 that the file carries. A reader that cannot MUST NOT present the residual spread as though it were the total.
degraded_intervals records the spans during which a stream's alignment did
not hold — a probe that stopped answering, a quality that fell below its
declared bound. A fact about an interval that has passed cannot be
reconstructed, so it is recorded when observed.
11. Session (0x06) — must-understand
Who and what this recording is of.
session_id : string // stable identifier for this session
study : string
participant_code : string // pseudonymous. §11.2
operator : string // opaque reference. §11.3
operator_asserted_by : u8
profile_name : string // the conformance profile declared, or ""
profile_version : u32
security_tier : u32
app_name : string
app_version : string
created_unix_ns : u64
devices : array<Device>
custom : map
Device {
device_ref : u32
kind : string // what sort of instrument
serial : string
firmware : string
clock_domain : string // which domain its timestamps are in
}
Must-understand: a reader that skipped this would present a recording with no statement of what it is of, which for a regulated record is not a degraded answer.
A file may hold more than one Session record, and the last one wins.
This is not permission for a writer to be careless. It exists because the first record is written before anything has been recorded, so it cannot state what only the finished session knows — when the recording was paused, how long it actually ran, which devices turned out to be connected. A writer that learns those things appends another record.
Earlier records are appended over, never amended in place. A session that was going to be one thing and turned out to be another has both statements in the file, in order, and the difference between what a study planned and what it did is often the interesting part.
Which record a reader takes depends on how it is reading, and the two answers agree:
- A sealed file carries the description in force in its summary (§4), which is where a reader looks. Sealing re-emits the current record there, so the summary holds exactly one however many the body carries.
- A file recovered from its checkpoints has no summary, so a reader walks
the records in order and takes the last it reached. A recording killed
between two
Sessionrecords therefore reports the earlier one, which is correct: the later statement was never made.
A reader must not treat a second record as an error, and must not merge fields across records: the record in force is the description, whole. Merging would let a field nobody restated survive out of a statement that was superseded, producing a description no writer ever made.
11.1 What a record must be able to answer about itself
Every regime that governs research records asks the same four questions:
- What produced this, and under whose authority?
- When, and by which clock?
- Has it changed — and if so, who changed it, when, and why?
- Can you demonstrate it has not changed otherwise?
Every one is a fact about an instant, and an instant that has passed cannot be re-observed. A missing answer can be filled only by fabricating it or by discarding the data. So the fields exist from the first record written, whether or not anything consumes them yet.
That is why §0.6 keeps the shape of the vocabulary fixed while the numbers stay free: a field that does not exist cannot be populated retroactively.
11.2 Participant codes are pseudonymous
participant_code is an opaque code. The mapping from code to person never
enters a CA3 file.
This is also what makes withdrawal tractable: one file is one participant's session, so honouring a withdrawal is deleting the file. §0.4 records why the format therefore carries no in-file erasure mechanism.
11.3 Identity is recorded, never authenticated
A CA3 file records the identity it was given. It does not verify it and it never claims to have.
An identity is an opaque reference that means something in the deployment's own system, carried beside an assertion method:
| Value | Means |
|---|---|
| 0 | unspecified |
| 1 | host-asserted — an application vouched for it |
| 2 | os-user — taken from the operating system |
| 3 | configured — read from configuration |
| 4 | unattested — no basis is claimed |
0 and 4 are different statements and a reader MUST NOT merge them.
unspecified says nothing was recorded about how the identity was established.
unattested says it was supplied and that no basis is claimed for it. A file
that writes the second where it means the first overstates its own provenance,
which is the one direction a regulated record must not lean.
A reader that meets an assertion byte this specification does not define MUST
read it as unspecified. These values are ordered by how much they claim, so
an unknown one falls to the weakest and not to the nearest — the same rule §5.7
states for schema conformance. Falling to unattested would turn a value a
later writer invented into a claim that writer never made.
An operator reference means this is who the application said was operating. It does not mean this is who was operating. That distinction must never be blurred, because it is exactly the distinction an auditor cares about.
This is the same pattern as an unbounded bias and as a missing alignment event:
the file states the quality of every claim it makes, and unattested is
conformant, honest, and visible.
authorisation_reference and its window on the Stream record say under what
authority a stream was captured, and for what period that authority held.
11.4 Conformance profiles
A session may declare a profile: a named set of assertions over what a recording contains. The file records which profile it was written under and whether it held throughout.
Whether a session conformed is a fact about that session. The report is presentation and can be built at any time; the record has to be captured while it is happening.
A profile MUST state, per assertion, whether anything enforces it. An assertion that is checked and one that is merely recorded look identical in a schema, and a reader who assumes a control behind a field that has none is worse off than one who was told there is none.
Non-conformance is reported, never silently corrected.
12. Annotations (0x45) — skippable
Trial, block, condition, and anything else an experiment is structured by.
The container names no key. Trial structure already exists as data — event messages carrying attributes. What the container adds is an index over the intervals during which an attribute held a value, and it records which keys it was configured to index.
Hard-coding trial_id here would put vocabulary inside the container, which
§5.7's compression rule forbids for the same reason.
keys : array<string> // the attribute keys indexed
values : array<string> // the distinct values observed, deduplicated
entries : { entry_size: u32, entry_count: u32, packed entries }
offset size field
0 4 key_index index into keys[]
4 4 value_index index into values[]
8 4 stream_ref the stream the events were on
12 4 reserved MUST be zero
16 8 start_stamp_ns in that stream's own domain
24 8 end_stamp_ns
32 8 first_chunk_offset
---------
40
Interval semantics. An entry is the interval during which
attributes[keys[key_index]] == values[value_index] held on stream_ref. A
writer opens an interval when an event carries that key, and closes it at the
next event on the same stream carrying that key with a different value, or
at the stream's last message. An event repeating the same value extends the open
interval rather than opening a second.
Derived and skippable. The events are the data; a reader that ignores this index gets the same answers by scanning, more slowly.
13. The remaining records
13.1 Footer (0x02) — fixed size
offset size field
0 8 summary_start 0 = no summary
8 8 summary_offset_start 0 = none
16 8 last_checkpoint_offset 0 = none
24 4 summary_crc CRC-32C over the summary section
---------
28
The Footer never grows. It is located by arithmetic from EOF, so a longer one would be unreadable by the reader that needs to read it. It is the only fixed-size record in the format and that is why.
last_checkpoint_offset lets a sealed-but-damaged file enter the checkpoint
chain directly instead of backward-scanning for a sentinel.
summary_crc covers summary_start up to summary_offset_start, or up to the
Footer's opcode byte when there is no offset section. The offset section is
excluded because it describes the summary rather than belonging to it, and the
Footer locates it directly.
13.2 Statistics (0x44) — skippable
Whole-file counts and per-stream timing observations. Skippable: a reader that ignores it reports the same data, having counted it itself.
Per stream, alongside §7.3's counters, the recorder's own timing observations — maximum inter-arrival gap, minimum observed transport delay, maximum queue depth — which are facts about the session that no reader can reconstruct afterwards.
File-level time bounds cover only streams in the file's own domain. Two domains' raw stamps are not comparable, and presenting a bound across them would be a derived figure presented as a measured one. This is correct, not a gap.
13.3 Metadata (0x40) and Attachment (0x42) — skippable
Free-form named key-value blocks, and files carried alongside the streams.
Attachment.data is bytes64, because an attachment may be a device-native
video larger than four gibibytes. Its CRC follows the data rather than preceding
it, so the record can still grow by appending fields.
A stated limitation: content_length precedes content, so an attachment
whose length is unknown before writing is unrepresentable. Version 1 requires
attachments to be stat-able. A live capture of unknown size is a message stream,
or a separate file.
13.4 SummaryOffset (0x48) and Padding (0x4A) — skippable
One SummaryOffset per summary group, naming its opcode, start, and length, so
a reader fetches one index kind in one range request. Padding is bytes to
ignore.
14. Extension and versioning
Four mechanisms, and each covers a case the others cannot.
1. Unknown opcodes. Every record is length-prefixed, so an unknown opcode is skippable if its range says so. §3.1 partitions the space so a record whose absence would change what the file says cannot be silently skipped.
2. Known records grow by appending fields. A reader ignores trailing bytes it does not recognise. Five records cannot grow this way, because their last field is not length-prefixed or their position is fixed:
Footer— located by arithmetic from EOFChunk— trailing records blob- the in-chunk message record — trailing payload
- any future erased-chunk form — trailing fill
- any record whose trailer is located from its end
3. Fixed-stride tables grow by entry_size. ChunkIndex, StreamDirectory,
Annotations, and Statistics.per_stream each carry their own. A later version
appends fields and raises the value; the leading bytes keep their meaning.
Three of the four are stride-addressable, and a reader seeks to
entry_size × n. StreamDirectory is not: its entry ends in a string, so
its entry_size covers the fixed prefix and a reader walks it (§7.3). The
growth rule is the same for all four; only the seek differs.
4. The message record grows through field_mask. Bits 8–15 are defined in
advance as exactly one varint each in ascending order, so an old reader skips
them and still lands on the payload boundary. Bits 4–7 are must-understand.
Cutting across all four: Header.required_features (§3.3), for a change
inside a known record that no skip rule survives.
Version numbering. The major version is the 0x31 byte in the magic.
Bumping it means an incompatible container and a reader MUST reject a file
whose magic does not match.
There is no minor version. A reader that must ask "which minor version" is a
reader that could have asked required_features, and the bitmap answers
precisely rather than by implication.
While §0.6's status is draft, none of this binds. These rules describe how
the format will evolve once it freezes; until then a change is an edit.
15. Reserved in version 1
An opcode allocated and semantics sketched, not implemented. A version 1 writer never emits one. §3.4 governs how a reader reports them.
| Opcode | Name | Class | required_features | Trigger |
|---|---|---|---|---|
0x0B | ErasedChunk | must-understand | — | A protocol recording two or more consenting participants into one session |
0x0C | Tombstone | must-understand | — | Same |
0x0D | StreamCrypto | must-understand | bit 1 | The first study profile requiring encryption at rest |
0x41 | MetadataIndex | skippable | — | Enough metadata records that scanning is material |
0x43 | AttachmentIndex | skippable | — | Same, for attachments |
0x46 | Integrity | skippable | — | The first deployment requiring tamper-evidence |
Assigned feature bits, and this list is the whole of them:
| Bit | Set when |
|---|---|
| 0 | At least one chunk uses zstd |
| 1 | StreamCrypto is in use |
| 2 | At least one ColumnarChunk (§6.5) is present |
Bits 3–63 are reserved and MUST be zero. Bit 1 is assigned to a capability §15 reserves and no version 1 writer sets; it is named here rather than left out so that a reader meeting it refuses it by name, which §16 item 2 requires and a bare bit number does not give. Bit 2 is set by a converter, and a reader that has not implemented §6.5 refuses such a file by name for the same reason.
Notes on three of them.
Erasure is reserved rather than dropped, against the dyadic case. A session recording two participants cannot honour one withdrawal by deleting the file. Nothing else in this document assumes erasure exists.
Encryption is the erasure that works. Overwriting bytes in place is not erasure on a copy-on-write filesystem, on wear-levelled flash, or in any backup. Destroying a key is. If both are ever built, encryption comes first.
That axis is closed now, and §7.5 is the record. MessageOffsets lets a
reader address a message inside a chunk directly. Without it §9's within-chunk
query is a linear walk, and a file-backed random read has to fetch the whole
chunk, because §6.2 gives a message no address and there is no smaller range to
ask a file for. measurement/baselines measured that at 8,612 MiB fetched for
ten thousand shuffled reads, against 8.5 MiB for the same reads over §6.5.
For a compressed chunk the blob is decompressed regardless, so the index saves parse work and no I/O; for an uncompressed chunk it saves both.
15.1 Unmeasured, and named as such
Nothing below has been measured. None of it may be quoted as a property of this format.
- The compression ratio of single-stream chunks against an equivalent heterogeneous file
- Selective-stream read cost in the regime §6's sentence describes — one stream holding ninety-five percent of the bytes. The reference recording's largest stream is 42.98 % and it carries no frame stream, so §6's "roughly two percent" remains a claim about a file that does not exist here. Closing this needs a fixture with a dense frame stream
- The cost of either durability policy in §8.5 on any real storage
- Scrub latency against a real viewer
Measured since 2026-09-09, and moved out of the list above rather than deleted, so that what was once unmeasured stays visible:
- Structural overhead on a real multi-stream session. 18.35 % of the
reference recording, of which 99.3 % is descriptor restatement — ten copies
of one 118,889-byte
FileDescriptorSet, being five schemas at §5's two appearances each. - The ownership half of §6's claim. Selecting a stream touches exactly its
own extents: 0.01 % of the file for
trial.events, 42.98 % forsensor.motion. What §6 omits is the structural floor a reader crosses whichever stream it wants, which is what makes the smallest stream cost 18.36 % rather than 0.01 %.
Both re-derive from docs/decisions/0007-the-container-describes-itself-and-its-records-are-protobuf.md, which states the commands and
reports every stream including the ones where selecting saves nothing.
15.2 Stated here, enforced by nothing
A rule nobody checks is documentation. §0.7 requires every gap to be named rather than discovered, and the rules below are ones this document states and no validator, reader, or test verifies. Each is either uncheckable in principle, or checkable and not yet checked — and which one it is is the useful half of the entry.
| Rule | Where | Why nothing checks it |
|---|---|---|
| A declared unit is the unit the numbers are in | §5.2, §5.4 | Uncheckable. A field annotated m carrying millimetres is well-formed and wrong; §5.2 says so in as many words. The floor buys uniformity of statement |
| A repeated field's declared ordering is the order it is in | §5.2 clause 4 | Uncheckable from the descriptor. No option carries an ordering; a schema states it in prose and a validator takes it on trust |
| A tensor form's components are in the order the producer packed them | §5.8 | Uncheckable. A transposed axis produces finite, plausible numbers. tensor_form now carries a declaration rather than a name, so the shape, the element type, the axis labels and the per-component units are all compared against the descriptor — this row is what is left after that |
| The numbers behind an invalid entry of a projection form are not read as measurements | §5.8, §5.6 | Uncheckable. body.tensor.v1 writes NaN into an untracked row, which its one packer guarantees and no form can. A projection form is assembled from fields any producer filled, so its invalid entries hold whatever was left there and the validity channel is the only thing that says so |
| A whole-field validity channel implies in both directions | §5.6 | Uncheckable. Whether true really means every element was measured is a fact about the producer, not about the schema |
| A per-element validity channel has one entry per element | §5.6 | Checkable, unchecked. A validator confirms the channel is bool or bytes and does not compare its cardinality against the covered field's |
| A coded value is an enum, or has a companion that resolves without an external table | §5.2 clause 7 | Checkable, unchecked. The clause is stated and nothing reads it yet. When something does, what it can check is the presence of an enum or of the named companion; that the companion is the useful one is a judgement no descriptor carries |
Every frame a schema names resolves | §5.5 | Checked, and currently failing — no record carries a declaration, so a file naming a frame cannot supply it. See §5.5 |
| A file carries declarations for the frames it names | §5.5 | Not expressible. Version 1 has no record a FrameCatalogue can ride in |
descriptor is the closure of imports and nothing more | §5.1 | Half-checked. A missing import is caught; a surplus one is caught only for a registered schema_id, by byte equality |
encoding names a format the reader can decode | §5.2 clause 1 | Checkable, unchecked by the schema validator, which is never handed the field |
| Schemas and streams appear in both the data section and the summary | §5 | Checkable, unchecked. Nothing compares the two copies |
| A stored mapping is what its observations imply | §10.7 | Checkable, unchecked. observations_stream and estimator_version exist to make the recomputation possible; no implementation performs it |
DeviceDelay and AlignmentEvent are not double-counted | §10.11 | Not reachable. Neither record is implemented, so §16 item 22's required fixture cannot be built |
A stream's degraded_intervals are surfaced with results inside them | §10.12 | Checkable, unchecked. The field exists and nothing populates or consults it |
| A per-stream sequence gap is not a loss unless the topic has one stream | §7.4 | Checkable, unchecked. Nothing counts streams per topic |
A stream's block_offset is the real distance to its tensor block | §5.9 | Uncheckable by the container. The distance is a fact about the payload's protobuf encoding, and this container never parses a payload — so it takes the producer's word, aligns that byte, and cannot tell a correct offset from one that points into the middle of a length prefix. A consumer testing the address it got sees a boundary either way. What catches it is a validator holding the descriptor beside the file, which is the same division of labour §5.2's floor already sits on |
| A file that declares alignment was written by a writer that honoured it | §5.9 | Checkable, and checked only for files this repository wrote. crates/container/tests/aligned_blocks.rs reads the offsets back out of the bytes rather than trusting the writer, but nothing validates a foreign file's declaration against its own offsets. ca3 validate is where that belongs and does not do it yet |
| A profile states, per assertion, whether anything enforces it | §11.4 | Not expressible. A profile is a name and a version; there is nowhere to record the answer |
15.3 Schemas with no tensor form
The canonical vocabulary is twenty-one messages. Five declare a tensor form and sixteen do not, and §5.8's test is what separates them: a form declares what the schema fixes, and a schema that fixes nothing about its array has none. This is the index, and the last column is §0.7's requirement that a deferral name the condition that reopens it.
# the twenty-one, and the five that declare a form
grep -rhoE "^message [A-Za-z0-9_]+" proto/ca3/canonical/v1/*.proto | wc -l
grep -rhoE "ca3\.options\.v1\.tensor_form\) = \{" proto/ca3/canonical/v1/*.proto | wc -l
The counts are of schemas; the table has one row per argument. Two rows name several schemas each, because one argument covers all of them, and a reader counting rows gets thirteen where the vocabulary has twenty-one. That ambiguity is not hypothetical: the sentence here read "five … and eight do not" until 2026-08-11, where eight was a row count, and two independent SDKs implemented the two readings — one exposing twelve no-form schemas and the other sixteen.
Every message in ca3.canonical.v1 is indexed, not only the ones a producer
usually announces. §5.1 defines schema_id as the protobuf fully-qualified
message name and places no further restriction on it, so EyeSample is as
announceable as GazeSample and a validator returns the same verdict for both.
A message absent from this index has neither a form nor a stated reason, and a
reader cannot tell refused from overlooked — which is the distinction §0.7
exists for. The set indexed here is exactly the set
ca3_canonical::descriptor::declared_messages returns.
| Schema | Form | Why, and what reopens it |
|---|---|---|
BodySample | body.tensor.v1 | [28, 8] float32, a packed block |
GazeSample | gaze.tensor.v1 | [3, 8] float32, three eyes projected |
ImuSample | imu.tensor.v1 | [6] float32, acceleration and angular velocity |
EegChunk | eeg.tensor.v1 | [channel, sample] float32 microvolts |
PpgChunk | ppg.tensor.v1 | [channel, sample] uint32 counts |
AudioFrame | deferred | Its element type is whichever of two values format carries. Reopened by a form able to name a field as its element type's source — which is one more assertion nothing could check, and buying an axis order with it is a trade this vocabulary has declined so far |
PointerSample | deferred | Two coordinate spaces either of which a producer may omit, and no validity channel to say which. Reopened by a validity channel on the coordinate fields, which is a schema change and not an annotation |
Projection2D | deferred | A [2] float32 form over point.x and point.y is writable today and would fix an order no two consumers could disagree about — a one-axis array of two components read by name cannot be transposed. The numbers that make the point usable are image_width and image_height, which are uint32; a form declares one element type, so it would carry the half that needs no declaring and leave out the half that does. Reopened by a form able to carry components of more than one element type — the same limitation AudioFrame meets from the other side |
VideoFrame | no | data is a bitstream under an encoded codec, and under VIDEO_CODEC_RAW the element type, the plane count and the dimensionality all come from fields |
DeviceState | no | Two of its numbers are vendor-assigned category codes, and a dense row is the shape that cannot say so |
Event | no | No numbers. The array is an encoding of an open vocabulary against a label set the study fixes, not the schema |
Flag | no | One truth value and its validity. A form would remove no choice two consumers could make differently |
KeySample | no | hid_usage is an identifier and modifiers is a bitfield; a dense row invites arithmetic on both |
Vector1–VectorN, Quaternion | no | Component types that fix no array of their own, and shape types besides: no element type, axis order or unit either. Vector3 is metres in body.tensor.v1 and dimensionless in EyeSample.direction, so a form on the type could have carried neither |
EyeSample, EegChannel, PpgChannel | no | Component types that fix no array of their own, and the form that covers their numbers already exists on the message that assembles them. gaze.tensor.v1 reaches EyeSample through its eye axis, eeg.tensor.v1 and ppg.tensor.v1 reach their channels through from_repeated: "channels". A form here would be a second declaration of one array, which is the disagreement §5.8 exists to prevent rather than a gap in it |
Nine of the sixteen are component types, and one sentence covers all nine: a
component type fixes no array of its own, because what a form declares — an
element type, an axis order, a shape — is settled by the message that assembles
the components and never by a component. The row order of gaze.tensor.v1 is
left, right, centre, and that is GazeSample's fact; the width of an
eeg.tensor.v1 row is EegChunk.samples_per_channel. The two rows above
differ only in what is left over after that: a shape type has no unit of its own
either, and a row type has a form already reaching into it.
Three of the sixteen are deferred rather than refused, and the difference is that a deferral names a change that would make the answer yes.
Where each argument is written. Seven .proto files carry the argument at
the message it applies to — audio, device, event, flag, key,
pointer, video. vector.proto carries the shape-type argument once, for all
five. The three row types are argued at the message that declares the form
reaching into them, in gaze.proto, eeg.proto and ppg.proto. Projection2D
is argued here and nowhere else, which is the one place this paragraph is not
yet true of the schemas themselves; moving it into gaze.proto changes the
descriptor bytes and so requires regenerating the conformance fixtures of §16,
which is why it is named rather than done quietly.
The corpus does not cover §16. §16 requires that every rule is exercised by a committed vector of each polarity. The corpus that exists varies one axis — the schema announcement — and none of §16's thirty-eight numbered items has a vector. Items 12 and 22 name their fixtures as ones that cannot be skipped, and neither exists.
16. Conformance
Four reader levels, each a strict superset of the one above. An implementation states which level it implements and MUST NOT claim a property of a level it does not implement.
Level 1 — minimal reader
- Rejects a file whose leading eight bytes are not the magic of §2.1.
- Reads
required_featuresat absolute offset 17 and refuses to open the file if any bit it does not implement is set, naming the bits. - Parses records as
<opcode><content_length><content>and applies §3.1: rejects an unknown must-understand opcode by name, skips and reports an unknown skippable one, and names a reserved opcode rather than calling it unknown. - Rejects a varint longer than ten bytes, and a non-canonical varint.
- Rejects a
stringthat is not valid UTF-8, and amapwhose keys are unordered or repeated. - Resolves
schema_refandstream_ref, and raises on an unresolvable reference rather than substituting a placeholder. - Decodes messages per §6.2, including the union rule, the
v = 0absent form on bits 1–3, both readings of a clear bit 3, rejection of any of bits 4–7, and skipping one varint per unknown set bit in 8–15. - Reports a file with no trailing magic as truncated, and never presents a recovered set as complete.
Level 2 — indexed reader
- Reads the Footer by arithmetic from EOF and locates summary groups.
- Strides every fixed-stride table by its own
entry_size, never a constant. - Implements §9.1 both paths, selecting on
chunks_overlap. - Passes a fixture with deliberately overlapping chunk ranges. A reader implementing only the ordered path returns plausible wrong answers rather than failing, which makes this the one test that cannot be skipped.
- Applies §7.4: never reports a per-stream sequence gap as a loss without first counting the streams on that topic.
- Implements §9.2, fetching no chunk belonging to an unrequested stream, and returning the same sequence for the same window every time.
- Applies §10.6's mapping and its inverse exactly, and labels every result with its basis.
- Encounters every record the data section contains, or states that it does
not. A summary-driven reader parses only the summary and jumps to chunk
offsets, so a file carrying an unknown must-understand record in the data
section reads as though it were not there — the silent failure §3.1 exists
to prevent.
required_featuresdoes not close this: it catches a new value inside a known record, not a whole record type that claims no bit. A Level 2 reader MUST either walk the data section's record spine at open, or state that it has not and then MUST NOT claim §3.1 conformance.
Level 3 — time-aware reader
- Reports the total uncertainty of §10.2, or reports that it cannot — and
never presents
residual_spread_nsas the total. - Recomputes each mapping from its observations stream and reports the file inconsistent when the stored mapping disagrees.
- Never describes a mapping as verified without having read its observations.
- Surfaces
degraded_intervalsalongside any result falling inside one. - Reports
UNKNOWN— not zero — for a device delay the file does not carry. - Applies §10.11: never adds a
DeviceDelayto a stream already covered by a trigger-derived mapping, and reports a file carrying both without a statement of which was used as inconsistent. A fixture carrying both is required, because a reader that double-corrects returns a plausible number rather than failing. - Carries the asymmetry of §10.5: reports a one-way mapping's bound as unbounded below and zero above, and MUST NOT collapse the two into a single symmetric figure.
Level 4 — recovering reader
- Implements §8.3 in full, including step 5: re-sorting by
(stream_ref, min_stamp_ns)and recomputingprefix_max_stamp_ns. - Validates every checkpoint candidate against all four checks of §8.2.
- Performs the forward seek-skip from the last checkpoint to EOF, so chunks written after it are not lost.
- Falls back to a full forward spine walk when the chain is broken, and says that it did.
- Detects a record whose
content_lengthruns past EOF and stops rather than reading past it. - Reports where a recovery walk stopped rather than rejecting the file. A truncated tail is bytes of unknown provenance; the strictness item 3 requires of a sealed file would throw away everything before the first oddity.
Writer conformance
- Never seeks backward during capture.
- Achieves §7.2's ordering invariant, or sets
chunks_overlapfor the stream that broke it — and never drops a message to preserve it. - Sets
field_maskas the union over the chunk's messages, and never fabricates a value for an absent field. - Sets
flagsbit 0 from the order it actually emitted, never from an assumption about its input. - Decides
required_featuresat open, per §3.3. - Restates every live definition in at least one checkpoint per cadence.
- Records both clock sources of §4.3, and the anchor pair, or leaves the anchor explicitly absent.
- Names no schema identifier in any compression decision; uses
Stream.flagsbit 0. - Declares its durability policy per §8.5.
The corpus
A conformance corpus accompanies this document. Two rules make it a test rather than a demonstration:
Both polarities are load-bearing. A valid vector asserts exact output. An invalid vector is well-formed bytes that violate a rule stated here, and the requirement is that an implementation surfaces the violation — normalising it into something valid is a failure, not a lenience. A corpus of only valid inputs cannot distinguish an implementation that checks from one that does not.
Every rule is exercised, and the corpus is committed. A rule no vector exercises is documentation, not a check. A corpus regenerated at test time agrees with the current implementation by construction and catches nothing; it is committed, and a separate test regenerates it in memory and fails on any byte difference.
Skipping is reported as a skip, never folded into a pass.