Sequencer Determinism: Why Wall-Clock Time and a Monotonic Counter Give You Two Different Order Books

Two NTP-synced clocks on the same rack can drift 50 microseconds apart. If the next two events in your order book land closer together than that, the record of which one happened first stops being something you know and becomes something you assumed.

Two fields get treated as one field far more often than they should: the wall-clock timestamp an event was tagged with, and the monotonic sequence number recording the position it was actually processed in. They usually agree closely enough that nobody has to think about the difference. When they do not agree, a replay sorted on the wrong field reorders trades that never reordered live, and a signal your desk trades on can flip sign with no code change at all.

Everything below comes out of a harness I built and ran. It generates a synthetic two-feed order flow, replays it twice through the same book code under two different ordering rules, and reports where the two books come apart. The generator is described in full further down, including the parameters that decide the answer, because quoting rates without them would be quoting an assumption as a measurement. Every number here is measured, including the ones that weaken the argument I started with.

Contents

• The Wall Clock and the Monotonic Counter Answer Different Questions
• How Accurate Is Clock Synchronization, Really
• What the Regulators Require, and What They Do Not
• What One Reordered Pair Costs
• I Ran the Gate: The Checkpoint Mismatch at Event 244,000
• The Bisect: Ten Probes to Event 243,155
• What the Gate Does Not Catch
• How the Answer Moves With the Clock Offset
• Does It Survive an Order-ID-Keyed Book?
• Deterministic Replay Is a Solved Pattern
• Three Recent Failures, Three Different Lessons
• Where This Technique Comes From
• Three Objections, Answered
• A Practical Framework: The Replay Integrity Check
• Conclusion

The Wall Clock and the Monotonic Counter Answer Different Questions

Every event that reaches your matching engine carries two different answers to the question “when.”

One is CLOCK_REALTIME: wall-clock time, synchronized toward UTC, and by definition adjustable. The Linux clock_gettime(2) man page states it directly. CLOCK_REALTIME “measures real (i.e., wall-clock) time” and is “affected by discontinuous jumps … and by frequency adjustments performed by NTP.” It can jump backward. It can be stepped by an operator. Its job is to tell you the calendar time of day, not to order two nearby events.

The other is CLOCK_MONOTONIC: a counter that only moves forward, immune to any adjustment made to the wall clock. The same man page says it is “not affected by discontinuous jumps” and “consecutive calls will not go backwards.” A gateway that stamps an incoming message with a monotonically increasing sequence number is recording the one fact a wall clock cannot guarantee: the actual order in which the gateway processed the message.

The confusion is not a careless mistake. A wall-clock timestamp is the field every log line has, the field every dashboard plots, the field a human reads without translation. A monotonic counter is an internal implementation detail, right up until the moment you replay two feeds tagged by two different clocks and need to know which event the book genuinely saw first.

How Accurate Is Clock Synchronization, Really

Ask how well-synchronized two clocks in the same data center are, and the honest answer spans three orders of magnitude, not one number.

Ordinary NTP is millisecond-class. AWS describes its own pre-2023 Time Sync Service in those terms: “customers have depended on the Amazon Time Sync Service to provide one millisecond clock accuracy.” A millisecond is twenty times looser than the 50-microsecond offset used in the replay run below.

Hardware-assisted PTP does far better, though not to zero. AWS’s enhanced Time Sync Service, built on PTP with hardware timestamping, is “designed to deliver clock accuracy in the low double-digit microsecond range within your instance’s guest OS software,” and AWS separately notes that hardware packet timestamping “improves event ordering, which can also be used to determine the actual order in which packets arrive,” which is precisely the property a wall clock alone does not supply.

Live measurements from two GPS-disciplined stratum-1 devices on the same subnet, published by the time-sync vendor Meinberg, show offsets of roughly 4 to 10 microseconds, but jitter of 14 to 151 microseconds on that single hop. Meinberg’s own guidance on hardware pulse-per-second signals is blunt: “a time offset error up to a few 10s of microseconds can be caused by interrupt latency” alone. And the reason PTP needs hardware timestamping in the first place is message-path asymmetry: without it, Meinberg notes, “the delay can be quite long, many microseconds, or even milliseconds.”

Put the range together. Millisecond-class on plain NTP. Low double-digit microseconds on a fully hardware-timestamped PTP deployment. Single-digit microseconds only under GPS discipline, with jitter still running into the hundreds. None of it is zero. The ambiguity window is a property of physics and network hops, not a symptom of a badly configured desk.

What the Regulators Require, and What They Do Not

Two regulatory regimes put a number on clock accuracy, and both get quoted more often than they get read carefully.

MiFID II RTS 25, Table 2, sets its tightest tier for “members or participants of a trading venue using a high frequency algorithmic trading technique”: a maximum divergence from UTC of 100 microseconds, with timestamp granularity of 1 microsecond or better. That is the HFT-specific tier. RTS 25 also carries 1-millisecond and 1-second tiers for other categories of activity, so quoting “the MiFID clock rule” without naming the tier gets the number wrong as often as it gets it right.

FINRA’s CAT rule is looser and scoped to a different population. Rule 6820(a)(1) requires that “each Industry Member shall synchronize its Business Clocks … at a minimum to within a fifty (50) millisecond tolerance.” That figure binds Industry Members: broker-dealers and trading firms. It does not bind the exchanges.

The exchanges answer to a tighter number. The CAT NMS Plan FAQ states that “the CAT NMS Plan requires Participants to synchronize their Business Clocks at a minimum to within 100 microseconds.” Participants are the exchanges themselves, and 100 microseconds is the same figure MiFID reserves for its HFT tier. The venue is held five hundred times tighter than the firm trading on it.

Neither standard addresses internal event ordering. Both bound divergence from an external UTC reference, for audit-trail reconstruction after the fact. A firm can be fully compliant on both counts, clocks synchronized within tolerance, timestamps granular to the microsecond, and still not know which of two of its own events happened first, because compliance measures distance from UTC, not the order two nearby events were processed in.

One gap is worth stating plainly rather than papering over with an invented case study: no SEC or FINRA enforcement action fining a firm specifically for clock-synchronization drift turned up in this research. The accurate framing is that the rule exists because the risk is real, not that a named firm was caught and fined for it. That may be the sharper reason to take it seriously, not the weaker one. Nobody publishes a post-mortem the day an imbalance signal quietly inverts and nobody downstream notices for a quarter.

What One Reordered Pair Costs

Order book imbalance, the ratio of resting bid depth to resting ask depth near the touch, is one of the most widely used short-horizon signals in market microstructure, precisely because it is cheap to compute and empirically informative.

Cont, Kukanov and Stoikov’s 2014 paper in the Journal of Financial Econometrics, using NYSE TAQ data across 50 US stocks, found that “over short time intervals, price changes are mainly driven by the order flow imbalance,” with “a linear relation between order flow imbalance and price changes, with a slope inversely proportional to the market depth.”

Linear is the word that matters, because a linear relationship does not fail gracefully. If the input flips sign the output does not get noisier. It points the other way, with the same apparent precision it had when the sign was right.

That is the theoretical stake, and it is the reason an ordering error is worth measuring rather than waving away. How much of it survives contact with actual numbers is a different question, and this article answers it further down with measurements rather than argument: how often the sign really inverts, how large the reading is when it does, and how long it stays wrong. The answer turns out to be smaller than this section implies, which is worth knowing before you spend an engineer’s month on it.

I Ran the Gate: The Checkpoint Mismatch at Event 244,000

The harness builds a synthetic two-feed event log, then replays it twice through one order book implementation: once ordered by the gateway’s monotonic counter (the golden run), once ordered by the host wall-clock timestamp (the candidate run). Every K events, both replays are reduced to a SHA-256 hash of the full book state, and the two hash sequences are compared.

Here is the generator in full, because every rate quoted below is a consequence of it rather than a property of any real market. Four hundred thousand events across two feeds, spanning 91.1 seconds of simulated time. The two feeds model two independent ingress paths into one gateway, two gateway processes or an A and a B line, not two different venues. That distinction matters later on. Inter-event gaps come from a two-mode mixture: one per cent of the time from 1 to 20 microseconds, a burst, and otherwise from 60 to 400 microseconds. That yields a mean gap of 227.8 microseconds, a median of 228.2, and roughly 4,400 events per second. Five price levels a side. Orders are added and cancelled against level depth, with cancel probability rising as a level fills, so the book holds a steady depth instead of growing without bound. Host B’s clock sits a fixed 50.0 microseconds ahead of host A. The seed is fixed, so the run repeats exactly.

Two features of that setup decide most of what follows, and both are worth naming before the results rather than after. Just over one per cent of consecutive events fall within 50 microseconds of each other, which is the entire population at risk. And the clock error is modelled as a constant bias rather than jitter, which is the easier of the two problems, because a fixed bias can be measured once and subtracted while jitter cannot. What follows is a floor, not a worst case.

Feed A produced 199,948 events, feed B 200,052. With B’s clock running 50.0 microseconds ahead, 2,020 cross-feed event pairs land inside that offset, and sorting by wall-clock time reorders 2,019 events relative to the order the gateway processed them in, half a per cent of the stream.

Every checkpoint through event 243,000 matches. At 244,000 the hashes split.

Every checkpoint through event 243,000 matches between the two replays. At event 244,000, the hashes split: eb957c4793... on the golden run against 4ecfa6e04f... on the candidate run. One earlier checkpoint mismatch in the same run healed by the following checkpoint. This one does not heal, which makes it the first checkpoint worth bisecting: mismatched, and still mismatched a thousand events later.

The full round trip, 400,000 events replayed twice and hashed throughout, completes in about a third of a second. That number deserves a caveat rather than a boast: the book here is five price levels a side, and hashing full state on a production book carrying order-level detail across hundreds of levels is a different workload entirely. What the timing establishes is narrow. The arithmetic is not what stops people running this check.

The Bisect: Ten Probes to Event 243,155

A mismatched checkpoint tells you a 1,000-event window contains a divergence. It does not tell you which event. Finding it by inspection means reading up to a thousand events by eye. Finding it by bisection means reading about ten.

The gate applies the same binary search git bisect runs against a commit history. Git’s own documentation describes the technique as “a binary search algorithm to find which commit in your project’s history introduced a bug.” Here the history is a checkpointed event stream, and the bug is the first event where the two orderings produce different book state.

Ten probes, each splitting the remaining window in half, converge on event #243,155.

Starting from the window (243,000, 244,000], the gate ran ten probes, splitting the remaining window in half each time and checking whether the golden and candidate book states matched at the midpoint. Mixed match and differs results converged on a single answer: the first differing event is #243,155.

At that event the two orderings disagree on which of two operations came first. The golden order, the gateway’s true monotonic sequence, applies event 243,154 first: feed B, an ADD of 100 shares at bid 99.96. The candidate order, sorted by wall-clock timestamp, applies event 243,155 first: feed A, a CANCEL of 100 shares at bid 99.96. The gap between the two events, measured on the wall clock, is 47.2 microseconds. The clock offset in this run is 50.0 microseconds. The pair lands inside the ambiguity window by 2.8 microseconds.

The two events sit at the same price level, the same size, and opposite operations: an add and a cancel of the identical 100 shares. Which one the gateway saw first determines whether 100 shares rest at that price or nothing does, because the cancel clamps at the floor of zero when it arrives ahead of the add. Most reordered adds and cancels in this run heal. Both operations still land, in either order, by the time the next event is processed, and the book ends up the same either way. This pair does not.

That mechanism deserves a boundary drawn around it, because it belongs to a particular kind of book. This one aggregates by price level, so a cancel is a subtraction and an over-subtraction clamps at zero, losing quantity that never comes back. Depth-aggregated books are not a toy choice, they are what most replicas built from a market-data feed actually are, but they are not the only choice, and an order-ID-keyed book behaves differently enough that it gets its own test further down.

The book state immediately after: bid depth of 2,900 on the golden run against 2,800 on the candidate run, ask depth flat at 4,600 on both, imbalance of negative 0.227 against negative 0.243. The sign has not flipped at this point. What has happened is that the two books stop reconciling. Event 243,155 is the last one at which they agree, and from there to the end of the run, 156,846 events or 39.2% of the stream, they never agree again.

Two clarifications on that figure, because it is the one most easily misread. It is not the first time the books differ: 614 earlier events differ transiently and heal, which is what brings the total count of differing events to 157,460. Those are swaps caught in flight, not a corrupted book. And 39.2% measures persistence, not magnitude. The difference is 100 shares at one price level, carried forward because nothing in the replay ever reconciles it. A book wrong by 100 shares for 156,846 events is a different problem from a book that is 40% wrong, and only the first one happened here.

What the Gate Does Not Catch

Checking hashes every 1,000 events is not the same as checking every event. That gap is where the most important finding in this run lives, and it is worth stating openly rather than glossing over, because it is more useful than a clean result would have been.

Walking the two replays event by event, not just at the 1,000-event checkpoints, and comparing the sign of the order book imbalance at every step, surfaces 85 events across the full 400,000-event run where the imbalance sign differs between the two orderings, roughly one in every 4,700. The first of those 85 is event number 3,761: imbalance reads +0.015 on the golden run and -0.015 on the candidate run.

The earliest sign inversion in the run, event #3,761, sits between two checkpoints that both match.

Event 3,761 sits between checkpoint 3,000 and checkpoint 4,000, and both of those checkpoints match. The hash at 3,000 is identical between golden and candidate. The hash at 4,000 is identical between golden and candidate. A gate checkpointing every 1,000 events reports two consecutive matches around this event and never learns it happened. The book reconverged by the time the next checkpoint was taken, the same way most of the 2,019 wall-clock reorderings in this run heal, but for a handful of events in between, the sign a desk would have read was inverted for exactly as long as it took to reconverge.

Two further measurements cut against the argument I started with, and both belong here rather than in a footnote.

The first is magnitude. That opening example, +0.015 against -0.015, sits close to the smallest disagreement in the run, so quoting it on its own flatters the case. Across all 85, the median absolute imbalance at the moment of disagreement is 0.050, the ninetieth percentile is 0.088, and the largest is 0.139. None reaches 0.20. Just over half sit at or above 0.05 and six per cent at or above 0.10. That is not noise, and it is not dramatic either.

The second is duration, and it is the one that changes the conclusion. At this clock offset, every one of the 85 disagreements lasts exactly one event. Not a few dozen, not a burst. One. The two books disagree on the sign, the next event reconciles them, and the episode is over. Across 400,000 events that totals 0.02% of the stream, and a signal computed on any aggregation window wider than a single event never observes it.

That retires a claim the source post made and this article inherited. “The number your desk trades on is fiction” is too strong for what the data shows at 50 microseconds. A one-event sign inversion is a real defect in the replay and close to invisible to a strategy sampling the book on any longer interval.

That qualifier carries the whole claim, so it is worth a paragraph rather than a footnote. Both of those measurements, the magnitude spread and the one-event duration, describe a 50-microsecond offset and nothing else. Re-run the same generator at 100 microseconds and they do not hold: 32,822 flipped events instead of 85, 8.2% of the stream rather than 0.02%, episodes running up to 13 events rather than uniformly one, and a largest absolute imbalance of 0.255 rather than 0.139. The de-escalation above is real for the regime it was measured in, and 100 microseconds is a regime the next section argues many readers are closer to than they think. Taking “small footprint” as the article’s finding, rather than “small footprint at this offset,” would be reading it exactly as carelessly as the claim it replaces.

There is a limit to the technique as well, and it is narrower than I first described it. A checkpoint interval bounds how much damage an undetected divergence can do before it is caught, in this run no more than 1,000 events of exposure at the outside, and it does not bound whether a divergence happens at all. That much holds. But the blind spot is a property of what gets hashed, not only of how often. Snapshotting full book state every K events is one design. Folding each event’s effect into a running accumulator, so a transient disagreement cannot heal out of the record before anything looks at it, is another, and it costs a fixed amount per event instead of a full state hash per checkpoint. Tightening the interval narrows the window. Changing what you hash is what closes it. This run implemented the first and not the second, which is why the 85 inversions above had to be found by a separate per-event walk rather than by the gate itself.

How the Answer Moves With the Clock Offset

A single run at 50 microseconds is one point. The question any reader actually has is where their own stack sits, so here is the same generator swept across clock offsets, everything else held fixed.

Clock offset	Events reordered	Share of stream	Sign disagreements
1 microsecond	0	0.00%	0
5 microseconds	416	0.10%	15
10 microseconds	949	0.24%	35
25 microseconds	2,015	0.50%	85
50 microseconds	2,019	0.50%	85
100 microseconds	25,486	6.37%	32,822
500 microseconds	269,856	67.46%	21,125

The same seven rows plotted. The curve leaves the floor at the tolerance the regulator permits.

This is a threshold, not a slope, and that is the useful part. Between 25 and 50 microseconds almost nothing changes, because the generator’s burst mode tops out at 20 microseconds: by 25 the offset already covers every burst pair, and widening it further catches nothing new. At 100 microseconds the offset starts eating into the main body of the arrival distribution, which runs from 60 to 400 microseconds, and the reordering count jumps more than tenfold. At 500 it takes two thirds of the stream.

Two consequences follow. The first is that you cannot interpolate your own position from someone else’s number. The step sits wherever your clock offset crosses your inter-arrival distribution, so two desks with identical clocks and different event rates are in different regimes entirely, and the only way to know which is to measure both numbers on your own stack.

The second needs stating carefully, because it is easy to over-read and I very nearly did. The jump lands at 100 microseconds, which is also the divergence MiFID II RTS 25 allows for high-frequency algorithmic trading and what the CAT NMS Plan asks of exchanges. That alignment is a coincidence of this generator rather than a fact about markets: the step sits at 100 microseconds because the arrival distribution’s main mode begins at 60, and moving that lower bound moves the step with it. Read as numerology it is worthless.

What is not a coincidence is the shape. There is a threshold, its position is set by your own traffic rather than by anyone’s rulebook, and a firm can sit exactly at the regulatory tolerance and land on either side of it depending on how fast its own book moves. Compliance is measured against UTC. Neither rulebook is measuring this.

One number in that table does not behave, and I would rather flag it than quietly drop the row: sign disagreements fall from 32,822 at 100 microseconds to 21,125 at 500, even as reordering doubles. Reordering and sign inversion measure different things, and at extreme reordering the two books are different enough that their imbalance readings stop straddling zero as often. That is a description of what happened rather than an explanation, and this run does not establish one.

Does It Survive an Order-ID-Keyed Book?

The divergence above rests on a cancel clamping at zero, which only happens in a book that aggregates by price level. A reasonable objection is that this is an artifact of the model: a real order-ID-keyed book rejects a cancel naming an unknown order rather than silently subtracting, so an add and an unrelated cancel would commute and the whole thing would heal. That is a checkable claim, so I rebuilt the book that way and ran it again.

The objection is right about the mechanism. What it led me to was more useful than either of us expected.

It is right that the flagship pair commutes. Under order-ID keying the add and the cancel at event 243,155 touch different orders, and reordering them changes nothing. Nearly all of the 2,019 reorderings become harmless in exactly that way.

What survives is much narrower. In the order-keyed run only two cancels were reordered ahead of the add of their own order. An order-keyed book cannot clamp, so what happens next is decided entirely by one policy choice: what the book does with a cancel naming an order it has never seen. There are two common answers, and they do not produce the same book.

Reject it, the way a matching engine answers a client, and the cancel is gone for good. The add arrives afterwards and rests. Nothing retries. Two orders that the golden run removed stay resting in the candidate book, and from event 39,632 onward the two books never agree again: 360,369 events, just over 90% of the stream, against 39.2% for the aggregate book.

Park it in a pending map keyed by order ID and apply it when the add lands, which is the ordinary handling in a replica assembling state from a feed, and the divergence disappears. Same log, same clock offset, same 1,952 reorderings: 982 events differ transiently while swaps are in flight, a quarter of one per cent, and the two books agree at the final event. Nothing permanent survives at all.

So the conclusion is not that one book design is safer than the other. The exposure lives in the orphan-cancel policy, and that policy usually gets chosen for reasons that have nothing to do with clocks. A replica that rejects orphan cancels because rejecting is what the venue does inherits a permanent corruption the venue itself cannot suffer, since the venue never sees a cancel before the order it names. That is worth knowing before the next replica gets written, and it is the one finding here I would not have reached without being told my first example was an artifact.

One thing the harness understates, in the order-keyed model’s favour. In a real feed handler, a cancel naming an unknown order is an integrity signal rather than a silent no-op, and a handler watching for it can raise it and resynchronise from a snapshot instead of carrying a phantom order forever. This harness has no resync path, so what it measures is the shape of the error and not its lifetime in a system built to notice. That is the genuine advantage of order-ID keying over depth aggregation, and it is not the one the objection started with: keying by order turns a silent quantity error into a loud, countable one.

Three caveats, because this is the newest part of the experiment and the least settled. The order-keyed generator works at order granularity rather than level granularity, so the two runs are cousins rather than a controlled comparison. A 90% figure resting on two events is mechanically sound, one phantom order corrupts a book permanently, but it is thin. And this generator assigns each order’s events to feeds at random, so an order’s add and its cancel can arrive on different feeds. At a single venue they would not: one order lives on one feed behind one clock, and cross-feed skew cannot reorder an order against itself. This failure mode belongs to a book assembled from two independent sources, not to a single venue’s own feed.

Deterministic Replay Is a Solved Pattern

None of this is a novel problem, and the production systems handling the highest event rates in finance solved it the same way: by refusing to let raw transport order anything.

LMAX’s Business Logic Processor, documented by Martin Fowler, “can handle 6 million orders per second on a single thread,” running “entirely in-memory using event sourcing.” A single thread processing a single, already-ordered stream sidesteps the wall-clock-versus-monotonic-counter question entirely, because there is only one place that gets to decide what the order is.

LMAX’s own writeup states the underlying reason more directly than anything else in this research: “Even with IP multicasting, replication is still needed because IP messages can arrive in a different order on different nodes. The leader node provides a deterministic sequence for the rest of the processing.” That is the same failure mode as the 50-microsecond drift above, one level up: multiple nodes each form their own view of what arrived when, and those views disagree unless one node is designated to settle it.

The same shape recurs everywhere this problem is taken seriously. Aeron Cluster requires deterministic command application against an identical starting state as the precondition for its replicas not diverging. Chronicle Queue states its design goal directly: “an append-only, zero-GC architecture durably stores every message on disk, ensuring rapid recovery, complete auditability, and deterministic replay.” kdb+’s tickerplant “writes all data to a tickerplant log (to permit data recovery),” and replay is sequential: “each list is read in turn and evaluated.”

None of these four systems trust raw transport, IP multicast, a shared feed, or parallel sessions to preserve order on their own. Each inserts one ordering authority instead, a leader, a tickerplant, a consensus log, and applies commands deterministically from an agreed starting state. The replay gate in this article is a small, personal version of the same discipline. It does not create determinism. It checks whether determinism, once assumed, actually held.

Three Recent Failures, Three Different Lessons

Public incidents from the last several years show three different ways this category of problem surfaces once it leaves a synthetic benchmark and hits production traffic.

Gemini’s ten-hour outage on December 10, 2021, is the clearest case of downstream state divergence. Gemini’s own post-mortem states that “on Friday, December 10, we experienced a service disruption that resulted in an exchange outage for ten hours” and that “the messaging system errors led to state divergence of some downstream systems.” Recovery required manually reconciling state across two trading systems, by hand, after the fact, exactly the forensic work a hash-checkpoint-and-bisect gate exists to make unnecessary.

Coinbase’s outage on May 7, 2026, teaches a different lesson and deserves precise framing. The exchange’s matching engine “runs as a Raft-based replicated cluster inside an AWS Cluster Placement Group,” and when “three of five matching-engine nodes went down and we lost quorum,” there was “no automated cross-zone failover.” Cancel-only trading resumed at 2:25 AM ET on May 8, full trading at 3:49 AM ET. This was an availability and failover gap, not an ordering bug, and it should not be described as one. Its value here runs the other way: a Raft-based cluster is exactly the single-ordering-authority pattern from the previous section, deterministic replication done correctly, and even a well-built deterministic system goes down hard when it loses the quorum needed to keep applying commands in an agreed order.

Base, Coinbase’s Ethereum layer 2, is the closest case to this article’s subject on June 25 to 26, 2026, though it needs one honest qualification before it can be used: a single sequencer has no two-clock ambiguity to begin with, so this is a state-hygiene failure rather than an ordering-under-clock-error failure. It demonstrates the consequence, not the cause. With that said, the mechanism is worth reading: “a bug in sequencer block-building logic that allowed stale journal state to persist after a transaction validation failure,” which “caused a sequenced block that contained an invalid state transition, leading to a block that could not be accepted by any other nodes.” A single sequencer is only as good as the state it carries forward. Stale state survived a failed validation, produced a block no other node would accept, and the chain fully halted, twice, 116 minutes and 20 minutes, from the same root cause.

Where This Technique Comes From

Hash-checkpoint-and-bisect is git bisect applied to an event stream instead of a commit history, and the broader practice it belongs to, deterministic simulation testing, has a documented decade of production use.

Antithesis, a company built around this style of testing, traces its own lineage: “practical adoption of this approach was pioneered at FoundationDB and Amazon Web Services around 2010,” with “one of the earliest recorded discussions of how to implement DST” being “Will Wilson’s talk at the Strange Loop conference in 2014.” FoundationDB’s own testing philosophy, run a deterministic simulation of the whole distributed system under a fixed seed, inject faults, and replay any failure exactly, is the same discipline this article’s checkpoint gate applies at much smaller scale.

TigerBeetle’s VOPR, its randomized simulation tester, is the closest structural match. “All non-deterministic parts of the system are stubbed out. This includes the clock, network, and disk operations,” and “the seed and Git commit hash can be used to replay back the exact simulation and bug.” Stub the clock, fix the seed, replay exactly, find the exact divergence. That sentence describes both TigerBeetle’s VOPR and the 400,000-event run in this article.

None of this is exotic tooling reserved for database companies. It is binary search, applied to state instead of code, run against a log you already have.

Three Objections, Answered

“Just use PTP.”

In its strongest form, this objection is correct as far as it goes. Hardware-timestamped PTP is a ten-to-a-thousand-times improvement over plain NTP, and the numbers above bear that out.

It is also not a complete answer. Full-fidelity PTP needs hardware support at every hop in the path, and a legacy switch, a virtualized instance, or a cloud region boundary anywhere along it breaks the chain. Production environments often include at least one of the three. Even a deployment that does everything right is documented as landing in the low double-digit microsecond range, not zero.

PTP shrinks the ambiguity window. It does not remove the category error underneath it: a timestamp, however accurate, still answers when an event was tagged, not the order your book actually processed it in. The two converge as accuracy improves, but they are never formally the same field, and the gap between them is exactly where the 47.2-microsecond pair in this article’s own run lived, inside a 50-microsecond offset that most deployments would already consider tight.

“Exchanges give me a sequence number.”

On a single feed, this is true and useful: a monotonically increasing sequence number gives a stable total order and lets you detect a gap the moment one appears.

It stops there. FIX’s own documentation frames sequence numbers as a gap-detection mechanism between two independent counters, one inbound and one outbound, and warns plainly that a “desync can lead to financial loss.” That is a statement about keeping one session’s counters aligned with the venue’s, not a statement about what happens to ordering once the feed fans out inside your own infrastructure, across parallel decoders, across replicas, across a book replay built from a recorded tape. There is no cross-venue sequence number at all: venue A’s message #100 and venue B’s message #100 carry no defined relationship to each other. The moment you combine two feeds, which the replay gate in this article does by construction, a single feed’s sequence number stops being able to answer the only question that matters: what order did your gateway see them in.

“Lamport clocks solve this.”

In its strongest form, a logical clock gives a mathematically correct causal order without needing a synchronized physical clock at all, which sidesteps the entire clock-accuracy discussion above.

It sidesteps the discussion by answering a different question. Aeron’s own documentation is explicit: “Lamport and Vector Clocks have no relation to a physical clock. They cannot be associated with wall time.” A Lamport tick tells you that event X happened before event Y inside your own system. It cannot be reported to a regulator as the time anything happened, and RTS 25 and CAT both require exactly that: a wall-clock time, within a stated tolerance of UTC. Logical clocks solve internal consistency. They do not solve the forensic requirement to state, afterward, what time something happened, and a firm that can prove correct causal order but cannot produce a compliant timestamp has solved half the problem.

A Practical Framework: The Replay Integrity Check

A checklist for anyone about to trust a replay, a backtest, or an OMS state machine on their own book.

1. Separate the two fields at ingestion. Never let a wall-clock timestamp silently become the sort key for anything you call the processing order. Keep the monotonic sequence number as its own field, all the way through.
2. Measure your actual clock offset. Do not assume the spec number. The NTP-versus-PTP figures above span three orders of magnitude, and Meinberg’s own live measurements show jitter running well past the documented number on a single hop. Measure your own racks.
3. Compare that measured offset to your book’s real inter-event gap. If any meaningful share of your cross-feed pairs land inside the offset, 0.51% in this article’s run, which produced 0.50% of events actually reordered, you have an ordering-ambiguity problem today rather than a theoretical one. Check where the step sits for your own traffic before deciding it is small.
4. Hash-checkpoint your own replay against a golden run. Any book you own qualifies: a replica, a replay harness, an OMS state machine. SHA-256 the full state every K events and compare.
5. On a mismatch, bisect. The same binary search git bisect runs against commits narrows a 1,000-event window to one event in about ten probes.
6. Do not read “no mismatch at the checkpoint” as “no problem.” This article’s own run needed a separate per-event walk to find its 85 sign inversions, every one of which lasted a single event and healed before the next checkpoint. If the signal matters more to you than the state, fold the signal into a running accumulator per event rather than sampling state at an interval. Changing what you hash closes a gap that tightening the interval only narrows.
7. Know where this stops working. A hash-checkpoint gate diagnoses a book you control. It cannot tell you whose clock moved on a shared sequencer or a tape you do not own. It can only tell you that the two orderings you fed it disagree.

Conclusion

Two fields, a wall-clock timestamp and a monotonic sequence number, usually agree closely enough that nobody has to think about the difference. This article exists because of the word usually.

Here is what the run showed, at the size it deserves rather than the size that would read better. One pair of events landing 47.2 microseconds apart, inside a 50-microsecond offset, left two copies of the same book disagreeing by 100 shares at one price level for the remainder of the replay. Across 400,000 events the imbalance sign came out opposite between the two orderings 85 times, each for exactly one event, at a median magnitude of 0.05. A real defect with a small footprint. I would rather publish it at that size than inflate it into a crisis.

The two findings I did not go looking for are the ones worth keeping. Swapping the depth-aggregated book for an order-ID-keyed one does not fix this. It makes the divergence permanent rather than partial, because a cancel rejected as naming an unknown order is never retried, and the order it should have removed rests for the rest of the session. And the exposure behaves as a threshold rather than a gradient: almost nothing happens until the clock offset crosses into your inter-arrival distribution, and on this generator that step lands at 100 microseconds, which is exactly the divergence MiFID II RTS 25 permits.

Two numbers decide which side of that step you are on, and neither of them is in this article, because both are properties of your stack rather than mine. Measure the real clock offset between the hosts feeding your book. Measure the inter-event gap distribution of your own cross-feed traffic. Put one against the other. If a meaningful share of your events land inside the offset, you have the ambiguity this article measured, on your own data, and the question worth funding is how often rather than whether. If none of them do, you can stop thinking about this entirely and spend the week on something that is actually costing you money.

---

Beyond this article, I advise a small number of trading firms on the architecture decisions that determine whether a book, a replay, or a backtest can be trusted under audit: sequencer design, replay integrity, and clock discipline across a multi-venue stack. If that is useful to you, a strategy call can be booked at hftAdvisory.com.

---

This article was originally shared as a LinkedIn post.