Skip to main content

What Ranvier is

Written for a researcher who runs studies with more than one instrument and has never used a data streaming framework. By the end you will know what problem Ranvier solves, roughly how it solves it, and whether it is worth reading further.

The problem​

You are running a study. An eye tracker samples at 200 Hz. A camera records the participant's face at 30 frames per second. A script decides when to show the next stimulus. A button box records responses.

Four instruments, and they all need to end up on one timeline.

The way this usually gets solved is that somebody writes glue. The eye tracker ships a Python library, so a script reads from it. The camera writes its own file. The stimulus program logs timestamps to a text file. Afterwards, another script tries to line all of it up.

Thousands of good studies have been run exactly this way. It also has four failure modes, and every one of them is quiet.

The glue is written once and cannot be reused. The script connecting this eye tracker to this stimulus program will not connect a different tracker to a different program. Adding a fifth instrument means editing code that already works, which is how working code stops working.

Timing is reconciled afterwards, from files. Each device stamped its data with its own clock, and those clocks disagree. Lining them up at analysis time means estimating offsets nobody measured while the data was being collected. If the estimate is wrong, nothing tells you.

Nothing is watching for loss. If the eye tracker drops fifty samples because a buffer overflowed, the file is fifty samples shorter and no marker says where. The gap is invisible in the data and invisible in the analysis.

The setup is not the record. How the pieces were wired together lives in somebody's memory, or in a paragraph of a methods section written later. A colleague cannot re-run it. Neither can you, eighteen months on.

How it works​

Ranvier's answer to all four is the same idea: the connections between programs are described outside the programs.

Each instrument gets a small program — a node — that does one thing: read the device and publish what it reads. A node never names another node. It publishes under a name, and anything that wants the data subscribes to that name.

What connects them is a text file:

ranvier_graph: 1
id: pointer-demo

nodes:
  - id: mouse
    type: devices-mouse
    publish:
      motion: pointer.motion

  - id: log
    type: recorder
    params:
      study: PROTO-7
      participant: P-014
    subscribe:
      input: pointer.motion

Two programs. The first reads a mouse and publishes under the name pointer.motion. The second writes what it receives to a file and subscribes to that name. Nothing in the file says "connect the mouse to the log." They use the same name, and that shared name is the connection.

That one move addresses the first failure and the fourth. Adding a live display means adding four lines to this file; neither existing program changes. And the file is not documentation of the setup — it is the setup, so it cannot drift from what actually ran.

The other two failures need more than wiring.

Every reading is stamped when it is published, against a clock chosen because it cannot run backwards, and carries the instrument's own timestamp alongside it untouched. The relationship between those two clocks is estimated while the data is being collected rather than guessed at afterwards, and how well that estimate is doing is recorded with it. spec/streams.md §4 is the whole of that story and it is the part most worth reading.

Every stream counts its own readings — 1, 2, 3 — so a gap means a dropped sample and cannot mean anything else. That is the difference between "the data looks sparse there" and "four samples were lost between 1.204 s and 1.224 s." One of those you can put in a paper.

Why it is this way​

Because the connections have to be data, not code. This is an old idea with a name — flow-based programming, described by J. Paul Morrison in the 1970s — and if you have used LabVIEW, Max/MSP, TouchDesigner, or Node-RED you have used it already. The property that matters: a program written this way can be dropped into a different study without being modified, because it was never told anything about the study it was in.

Because Ranvier decides what a reading looks like, not the manufacturer. Two eye trackers measure the same thing and report it differently — different field names, different units, different ideas of where the origin is and which way the axes point. If each integration publishes its own shape, every analysis program needs a branch for each tracker, and the coupling has just moved from which program to which format. So for anything many instruments produce — gaze, hand tracking, body pose — Ranvier defines one schema and integrations conform to it. Swapping trackers becomes one line in the file above.

Because losing data quietly is the worst thing a research instrument can do. Most of the design follows from preferring a loud failure to a quiet one. A gap you can count beats a gap you cannot see, and a session that stops and says why beats a session with a hole in the middle nobody noticed.

What it costs​

A graph is ceremony for two programs. If you have one instrument and one script, a direct socket is less work and always will be. Ranvier earns its keep at three instruments and above.

Nodes are separate programs, so a message between them is encoded and decoded. That is real cost. It buys the ability to write a node in any language, and crash isolation, and it is affordable at the rates research instruments actually produce — see spec/wire.md.

Canonical schemas mean a device that does not fit loses something. If a tracker measures a quantity the schema has no place for, it is dropped or the schema has to widen. Interchange is bought with a little information.

Ranvier cannot guarantee a deadline, and does not pretend to. Anything needing genuinely sub-millisecond timing belongs in dedicated hardware, which Ranvier orchestrates and records around rather than replaces.

Where to go next​

  • spec/wire.md — the bytes two processes exchange, and the topology they exchange them over. The next thing to read.
  • spec/streams.md — sequence numbers, gap accounting, clocks and timestamps. §4 is the hard part.
  • spec/node.md — what it takes to be a node: the manifest, the ports, the lifecycle.
  • Glossary — if a term is unfamiliar.