Designing Market Data for Deterministic Replay

Ariel Silahian

Ariel Silahian is a senior technology executive in institutional electronic trading, with 30+ years across the buy and sell side (New York, Miami, London, Hong Kong). He is the author of "C++ High Performance for Financial Systems" (Packt) and the creator of VisualHFT, the open-source microstructure analytics stack. He writes on exchange architecture, market microstructure, and execution quality, and advises a select number of trading firms on infrastructure decisions that move P&L. Talk architecture: https://hftadvisory.com

You are designing the market-data path for a trading system. Somewhere in the requirements, written down or not, is one that quietly shapes every layer beneath it: any moment the system traded through has to be reconstructable later, exactly. Feed the system the same input it consumed at 09:31:07.442, in the same order and with the same relative timing, and it should produce the same output down to the bit.

That property is deterministic replay. Most designs treat it as something to add when an incident finally forces the question, and by then it is already gone. Replay is a property of the market-data path you design in from the start or lose. This is a walk through the design decisions that keep it, in roughly the order you make them.

Table of Contents

1. The Requirement Most Designs Skip
2. Know What Breaks It: The Six Determinism Leaks
3. Decision 1: Capture at the Boundary
4. Decision 2: Event Order Belongs to the Data, Not the Scheduler
5. Decision 3: Replay Is a Runtime Mode
6. Decision 4: Verify Fidelity, Then Size the Storage
7. Porting the Design to Other Venue Classes
8. When the Design Is Done

The Requirement Most Designs Skip

Write the requirement down before you draw the first box, because it constrains all of them: for any past interval, you can produce the exact input the system consumed, and re-running the production code over that input reproduces the production output.

Two halves, both load-bearing. The input has to be the real input, captured as it arrived, rather than a later approximation rebuilt from something downstream. And the code path over it has to be the real production code path, not a separate model of it. Miss either half and replay becomes a plausible story you cannot verify, which is worse than no replay at all because it looks like a control while behaving like a guess.

Everything that follows is in service of those two halves. Capture the real input at the earliest point you own. Fix the order of that input in the data itself. Run the same binary over it. Then prove the output matches.

Know What Breaks It: The Six Determinism Leaks

Determinism breaks at specific, findable points, and your design has to close each one, so name them first. Each leaves a distinct signature you can look for in a system that has the leak.

Wall-clock reads in the hot path. Any branch that reads the system clock as part of a decision, rather than only for logging, injects a value that differs every run. Signature: feed the same recorded packets through the strategy twice and get two different decisions. The design rule is to separate clock reads used for timestamping, which are necessary, from clock reads that feed logic, which are a leak.

Non-deterministic thread scheduling in a multi-threaded merge. When several threads each consume a channel and hand events to a shared merge point, the order the OS schedules them is not guaranteed run to run. Signature: a nightly regression replay produces a slightly different book state each night on identical input files, with no code change.

Unordered UDP multicast arrival. Exchange market data typically arrives over UDP multicast with no transport-layer ordering guarantee, often over redundant A and B lines delayed independently by network path. Signature: two packets carrying the same event arrive in a different relative order depending on switch load or NIC queue, and merge logic that trusts arrival order instead of sequence number builds a different book on different days.

Gap-fill and retransmission races. When a gap is detected and a retransmission or snapshot-recovery request goes out, the recovered data arrives interleaved with live traffic. If the splice point is decided by arrival time rather than sequence number, the resulting book depends on network timing that varies run to run.

Floating-point non-associativity. IEEE 754 arithmetic is not associative: summing the same values in a different order can produce a different result. If a book aggregate, a VWAP, or any derived quantity is computed over an event order that is not itself fixed, the value is not reproducible even when the underlying facts are identical.

Any I/O not captured at the boundary. A config read, a reference-data lookup, any source the strategy consults that is not part of the recorded capture is an escape hatch. Replay looks clean until one of them silently returns a different value than it did in production.

Six leaks, four design decisions. The rest of this closes them.

Decision 1: Capture at the Boundary

Record the raw wire feed at ingress, before parsing or normalization touches it, with a hardware timestamp applied when the packet arrives rather than a software timestamp applied whenever the application thread got to it. Timestamping at ingress in hardware is an architecture decision you make once and inherit everywhere downstream: put the clock discipline upstream of any software jitter, or every timestamp below it carries whatever jitter the software layer added. Precision Time Protocol (IEEE 1588) is the standard built for this, targeting sub-microsecond synchronization.

This is where the distinction between ground truth and re-derived state becomes load-bearing.

The raw bytes captured at ingress are ground truth: exactly what arrived, in exactly the order and timing it arrived. The book your production system built from those bytes, and the book you rebuild from a later replay of them, are both derived artifacts, and they can diverge for reasons that have nothing to do with the market: a parsing bug, a gap-handling decision, a message dropped and never retransmitted. Capture only the derived snapshots and never the raw bytes, and you cannot tell whether a discrepancy is a market event or a bug in your own book-building logic.

This also settles a tempting shortcut. If you can reconstruct from the exchange’s official consolidated record, why capture your own feed at all? Because the consolidated record is not what a fast system saw. Work comparing direct exchange feeds to the consolidated SIP tape found direct feeds arriving on the order of tens of microseconds after emission, versus roughly 1,128 microseconds via the consolidated tape, a gap of more than 50 to 1. By the time the official record shows a price, a fast system has already acted on a book state that record does not contain. Capture at your own boundary, or reconstruct a market your system never traded in.

Decision 2: Event Order Belongs to the Data, Not the Scheduler

If event order depends on which thread the OS scheduler happened to run first, it is a property of that day’s scheduler state rather than of your system. Make ordering a property of the data itself, decided once by a single deterministic authority, rather than an emergent outcome of concurrent writers racing to a shared structure.

This is the single-writer principle, from Martin Thompson’s work on the LMAX Disruptor. It establishes a deterministic total order for writes among concurrent producers, removing the ordering ambiguity that exists when multiple writers contend for the same structure. It does not, on its own, eliminate scheduling nondeterminism on the consumer side, which is a separate concern downstream of the write. As a reference for the cost, LMAX Exchange has published figures of over 25 million messages per second at latencies under 50 nanoseconds, so a deterministic write order is not the throughput sacrifice people assume.

In the market-data path this means: when you combine feed channels, redundant A and B lines, multiple instruments, multiple venues, into one ordered stream, assign a single deterministic merge authority to decide the final order rather than letting whichever thread reads a packet first win the race. That authority orders strictly by the sequence discipline the data carries, the per-channel sequence numbers exchange protocols are built around, never by wall-clock arrival.

This is event sourcing in the sense Martin Fowler describes: every state change captured as an event, stored in the order it was applied, which is what makes reconstructing any past state by replaying those events in that exact order possible at all. A path that has not fixed its event order has not implemented event sourcing whatever it stores; it has events with an ambiguous order, a materially weaker guarantee.

There is a test that proves the design works. Replay the same captured packets through the merge logic several times, injecting artificial delay at different points each run, and confirm the resulting event order is byte-identical every time. If it is not, the merge order is still a property of the scheduler, and you have found the leak before it found you.

Decision 3: Replay Is a Runtime Mode

Run replay through the same binary that runs in production, not a separately built simulator meant to approximate it. A simulator maintained as a second codebase drifts from production the moment either one changes, silently, and nobody notices until the replay result and the production result disagree for a reason that has nothing to do with the incident you were investigating.

So design replay as a runtime mode, a swap of the input source, not a separate program. In production the ingest layer reads live multicast; in replay it reads recorded capture. Every layer between ingest and decision, the parser, the book builder, the strategy logic, the risk checks, neither knows nor cares which source it reads from, and produces the same output either way, because it is the same code.

This is the same discipline that database engineering arrived at independently. FoundationDB was built entirely inside a deterministic simulation for eighteen months before its team ever ran it against real infrastructure, using a seeded deterministic random-number generator so any failing run reproduces bit-for-bit from its seed alone. Testing the real execution path under controlled, reproducible conditions rather than a simplified model of it is the principle; a market-data path that replays through its own production binary is that principle applied to trading.

Decision 4: Verify Fidelity, Then Size the Storage

Replay capability that has never been checked against production output is a hypothesis, not a control. Build the check in: run the replay and compare its full output, or a checksum of the state at each decision point, against the production output recorded from the live run. Any divergence at the byte or state level means one of two things, and both are useful. Either the replay path is not faithful to production, or something in the live path was never captured, and you have found the next escape hatch before it caused an incident.

Fidelity is worth more than debugging. Every microstructure signal a desk runs, VPIN, order-book imbalance, effective spread, adverse-selection markouts, is computed off a reconstructed book. If that book was rebuilt through a path you cannot replay deterministically, the signal’s history is unauditable: you cannot prove the number your strategy acted on is the number your research would reproduce from the same capture. Replay fidelity is what makes a microstructure metric trustworthy enough to trade on rather than only to plot.

Then size the storage, because full-fidelity capture has a real cost and the design has to account for it. A 10 Gbps feed near line rate generates on the order of 4.5 terabytes per hour of raw packet data. That is a retention-duration decision, not a wall. The minimum viable version is capture at a single boundary with a hot-retention window sized to your incident-discovery lag, with older capture moved to cheaper cold storage, rather than an appliance-grade deployment across every feed. Where the line sits is set by what a book divergence actually costs you: an incident found three days after the fact needs three days of retained capture at reconstruction fidelity, so a rolling buffer sized to “today” loses the ability to answer the question on day four.

Porting the Design to Other Venue Classes

The four decisions are venue-agnostic, but what counts as ground truth is not, so the design ports with one substitution. On a public blockchain, canonical state is probabilistic until finality: a reorg can rewrite the block your system already treated as settled, so the ground truth you capture is your node’s observed view, not a single wire feed every participant shares. Sequence discipline comes from block height and per-exchange message sequence numbers plus cross-exchange clock skew rather than a PTP-synchronized multicast feed. The substrate changes; the four decisions do not. Capture at your own boundary, fix order in the data, replay through the same binary, verify the output.

When the Design Is Done

The design is done when the team can answer these six with specifics, not with “we’d have to check.”

• Can you name the exact bytes the strategy consumed for any past millisecond, or only the log line it produced?
• Is replay driven by the same binary and hot path as production, or by a separately maintained simulator nobody has verified against production output recently?
• Can you prove replay output equals production output with a checksum or a state diff, or does “it looks about right” pass for verification today?
• Where in the path is the first non-deterministic read, wall clock, thread interleaving, unordered arrival, and can the team name its exact location without going to find out?
• How far back can you actually replay, and what does a day of raw capture cost you given your real incident-discovery lag rather than an assumed same-day discovery?
• When the A and B lines disagree, or a gap-fill races live traffic, is the resolution deterministic and repeatable on a second run, or dependent on which packet arrived first that day?

One case stays hard even after all four decisions are made well: cross-venue reconstruction, when an incident’s causal chain crosses two or more venues, each with independently captured clocks and independently designed capture boundaries. The single-venue design above solves the single-venue case cleanly and does not solve that one. If you are designing for a desk that trades across venues, that is the part of the design to pressure-test first, because it is the part the rest of the architecture will quietly let you believe is finished.