The Anatomy of a $50M Latency Build: Trading System Architecture Decisions That Separate a Machine From a Money Pit

Ariel Silahian

Ariel Silahian is a senior technology executive in institutional electronic trading, with 30+ years across the buy and sell side (New York, Miami, London, Hong Kong). He is the author of "C++ High Performance for Financial Systems" (Packt) and the creator of VisualHFT, the open-source microstructure analytics stack. He writes on exchange architecture, market microstructure, and execution quality, and advises a select number of trading firms on infrastructure decisions that move P&L. Talk architecture: https://hftadvisory.com

Table of Contents

• Introduction
• Failure 1: Speed Bought Before Measurement
• Failure 2: The Critical Path Nobody Sequenced
• Failure 3: Rebuilding What Should Have Been Licensed
• Instrument First: The Two Measurements That Keep a Build Honest
• The Build Order: Sequence the Critical Path, Keep the Rest in Cloud
• Build vs License: Where the Edge Lives vs Where Milliseconds Are Tolerable
• The Instrument-Before-Spend Sequence
• Conclusion: The Capital Call That Isn’t Mine

Introduction

Investors planning an independent quant fund keep asking me the same question: what does a latency-sensitive trading system architecture actually cost to build. I answer with a composite drawn from the pattern I have seen, not a single named fund: $50M burned in the first two years, the strategy still without an edge to show for it. The investor’s read is almost always the same: low latency is a money pit. Mine is different. The burn has an anatomy, and every line of it is architectural, a decision someone made or failed to make before the first server rack shipped. Underneath all three failure modes in this article sits one decision that matters more than any other: whether each component gets built in-house or licensed. That single call decides how much of the budget reaches the strategy versus how much reinvents plumbing someone else already sells. I return to it with a worked example later.

This applies well beyond a greenfield build. I get the same question from a CTO modernizing a stack built ten years ago, and from a growth-stage founder several raises away from ever seeing $50M whose real question is how not to spend it and still compete. Instrumenting a system you already run finds the tail you are already paying for without a rebuild, so the return is often higher on the existing stack than on the new one. It holds whether the fund trades TradFi futures out of a Chicago colo or runs a multi-venue crypto book. The venue changes; the failure pattern does not. The $50M disappears the same way every time: bought before it was measured, built in the wrong order, and rebuilt where it should have been licensed.

I am the architect retained to hold a build to discipline before the spend, not the engineer who writes the feed handler; I work with specialized engineering teams on implementation and stay above that layer. What follows is the three decisions that determine whether your budget builds a machine, whether the build is new or the one you already run, the conversation that happens before the vendor calls start.

Failure 1: Speed Bought Before Measurement

The first failure mode is the most common and the most expensive, because it compounds. A desk buys speed before it has measured anything. Colo space gets leased, a low-latency network gets provisioned, premium hardware gets racked, and the system goes live with software timestamps as the only record of how fast anything actually happened.

Software timestamps are stamped by the application, after the packet has already crossed the OS network stack, however many context switches the kernel scheduled that microsecond, and whatever jitter the CPU’s own load introduced. They tell you when your code noticed the packet, not when it arrived. The difference between those two numbers is exactly the latency the desk just spent seven figures trying to buy, and nobody can see it.

The production signature is diagnostic silence. When a strategy underperforms, nobody can say whether the problem is the signal, the risk gate, the exchange, or the network path, because there is no tick-to-trade baseline to test any hypothesis against. Every postmortem becomes an argument about intuition. Regulators drew this line years ago. Under MiFID II RTS 25 (EU Delegated Regulation 2017/574), participants running gateway-to-gateway systems at one millisecond or less must keep clock divergence from UTC under 100 microseconds, with timestamp granularity of one microsecond or better. That is a compliance floor, and it exists because anything coarser cannot reconstruct what happened in the order flow. A desk trying to prove its own edge with software timestamps is holding itself to a lower bar than the regulator does.

The fix is sequencing. Instrument before you spend on speed, so every subsequent dollar has a before-and-after number attached to it. Speed bought blind is a bet. Speed bought against a baseline is an investment with a return you can report to your board or LPs.

Failure 2: The Critical Path Nobody Sequenced

The second failure mode shows up in the build plan itself, or its absence. Funds under time pressure build everything at once: the multicast feed handler, the order book, the signal engine, and the exchange gateway, all in parallel, racing toward a single go-live date. It feels efficient. Four teams, four workstreams, one deadline.

It is the opposite of efficient, because these four components form one critical path with a fixed order: the feed handler’s output shape determines what the book can consume, the book’s update semantics determine what the signal can safely read, and the signal’s output timing determines what the gateway must guarantee under load. Build them in parallel without a tested contract between each stage, and you get integration failures precisely where the microseconds live, discovered under production load instead of in a design review.

The cost of shipping an unsequenced, undertested critical path is real and well documented, and the canonical example is canonical because it actually happened. Knight Capital, August 1, 2012, is usually told as a deployment-discipline story, and that is the correct proximate cause: an undetected deployment error ran for roughly 45 minutes, generated around 4 million unintended orders, left a roughly $7 billion position, and produced a loss of about $440 million. The reason it belongs here is narrower. An unsequenced, ungated critical path is what turns a deployment error into a $440 million one, because sequencing and gating bound the blast radius when a deploy goes wrong, and one eventually will. A path built in order, each stage tested against the prior stage’s real output with a risk gate inline, contains a bad deploy to a recoverable incident. An unsequenced one turns the same deploy into a solvency event.

The desks that get this right build in order. Feed handler first, proven against real market data. Book second, proven against the feed handler’s actual output, not its spec. Signal third, proven against the book’s real latency profile. Gateway last, proven under the signal’s actual load, with the risk gate already in place. Slower to first light. Far cheaper to keep alive.

Failure 3: Rebuilding What Should Have Been Licensed

The third failure mode is the quietest, because it looks like diligence rather than waste. A desk builds everything in-house on the theory that owning the full stack means owning the full edge. In practice, engineering hours go into rebuilding market-data normalization, reference-data plumbing, and other commodity infrastructure a dozen vendors already sell at production quality, while the signal that is supposed to generate the return sits under-resourced and behind schedule.

I have seen this inversion in almost every build I review, TradFi and crypto-native alike: a normalized multi-venue feed rebuilt from scratch while the signal that justifies the spend still runs on a spreadsheet, or a FIX order-routing layer rewritten for control that never gets exercised because nobody has time to touch the code once the signal work falls behind. Even at the protocol layer this shows up. Whether a desk parses verbose FIX messages or a binary protocol like CME’s iLink 3, production reliability comes down to discipline, not encoding format: a well-run FIX gateway from a proven vendor beats a hand-rolled binary decoder maintained by an engineer who has since moved to the signal team.

The discipline is asking, before a line of infrastructure code gets written, whether this component is where the fund’s edge actually lives. If not, license it and put the freed hours where the edge is. I walk through a worked version of that tradeoff, with representative numbers, in the build-vs-license section below.

Instrument First: The Two Measurements That Keep a Build Honest

Every fix above routes through the same starting point: instrument before you spend, and keep instrumenting after. Two measurements do almost all of the work.

The first is wire-to-wire tick-to-trade latency, captured with hardware timestamps at the network card rather than software timestamps in the application. Hardware timestamping removes the OS stack, the scheduler, and the application’s own jitter, so the number reflects what happened on the wire, not what your code noticed. STAC-T1, the industry benchmark developed with CME and a proprietary trading firm, defines tick-to-trade as exactly this: from receiving a price update on the market-data feed to transmitting an order back to the market. Public results show the scale well-instrumented systems reach: Redline reported a mean of 6.1 microseconds on STAC-T1.EMINI, a maximum of 18.2 microseconds, and 1.4 microseconds of jitter. I cite that for the tails, not as a target. The mean tells you almost nothing on its own. What matters is the distribution at p99 and p99.9, because averages hide exactly the tail events that cost money: the rare, ugly spikes that hit when the market is moving and the fill matters most.

Where you plant that instrument depends on the venue. For a TradFi futures desk, the wire terminates at a card you control in a rack you leased. For a crypto-native fund on a CEX, it frequently does not: the API endpoint sits inside the exchange’s own cloud, and the exchange’s internal matching and gateway latency is a floor you cannot shave. The discipline is unchanged, only the instrument point moves. Measure wire-to-wire from the furthest point you control, the exchange’s public API or its lower-latency access tier, treat everything behind that boundary as fixed cost, and hold your own code to the same p99/p99.9 standard on your side of the line.

The second measurement is adverse-selection markout at the holding horizon. Markout is the price movement over a fixed interval before and after a fill, the cleanest proxy for whether the fund is getting picked off by faster or better-informed counterparties. The horizon is not cosmetic. Too short a window systematically understates adverse selection, because a toxic fill’s damage often shows up several holding periods later, not in the next tick. A fund checking markout at one second when its holding horizon is minutes will call toxic flow clean. Together these turn “the strategy underperformed” from opinion into diagnosis: tick-to-trade tells you whether the infrastructure is the bottleneck, markout whether the fills themselves are toxic. Both are the baseline every subsequent infrastructure dollar is justified against.

The Build Order: Sequence the Critical Path, Keep the Rest in Cloud

Once the baseline exists, the second decision is sequencing the build, translating the critical path from Failure 2 into a deployment order. The critical path, feed handler to order book to signal to gateway, belongs in colocation, built and tested in that sequence, with kernel bypass networking to remove OS network-stack overhead and support line-rate processing at 10 gigabits per second and above, and pre-trade risk checks inline on that path rather than bolted on as an external async check. That is the entire footprint that needs to sit in colo. Research, backtesting, and model development, anything not on the order’s route to the market, belong in cloud, where iteration speed and cost matter more than microseconds.

The split is a judgment call, not a formula, but the pattern holds across venue classes: colo for the path where microseconds are the product, cloud for everything else. For a crypto-native market maker it maps onto exchange access tiers rather than one physical facility: co-locating within a major CEX’s matching-engine cloud region for the two or three venues where the spread genuinely depends on it, serving the rest over standard connectivity. Rate-limit-aware order routing and cross-venue self-match prevention belong on that same critical path, built and tested with it, because a throttled or rejected order under load is the same production failure as a slow one. And unlike a futures desk whose venue closes overnight, that path runs 24/7, so the monitoring and gating around it run continuously too.

Getting the split wrong is expensive in both directions. Put research in colo and you pay colo rack rates, mid-five to mid-six figures annually per venue by practitioner estimate, for workloads that gain nothing from proximity. Leave a real critical-path component in cloud, or serve it from outside the exchange’s region when that region is where the spread lives, and you reintroduce exactly the latency variance the build was meant to eliminate. There is also a timing dimension funds underweight. Colocation capacity at the venues that matter has been tightening in every recent build I have reviewed: pre-lease conversations that used to take weeks now routinely take months, so a badly sequenced or late colo decision burns lead time on a resource that may not be there on the next request.

Build vs License: Where the Edge Lives vs Where Milliseconds Are Tolerable

There is a real argument that speed itself is no longer the differentiator: speed has become the floor, and the edge has moved to the decisions around it, what to build, what to buy, what to consolidate. It is directionally correct and understates its own conclusion. If speed is the floor and everyone there is roughly equally fast, the build-vs-license decision becomes the edge itself rather than a cost-optimization exercise beside it. The sequencing and sourcing decisions in this article are the moat for a build under real capital constraints, because they determine how much of a limited engineering budget reaches the signal versus how much is spent reinventing infrastructure someone already sells. Latency variance is the canary here: an unstable tick-to-trade distribution at p99.9 is the tell that something underneath the speed number is architecturally unresolved.

The threshold I apply, as practitioner judgment rather than universal law: build raw feed handlers and normalization only where the edge genuinely lives, the component the signal depends on for its differentiated information, and license normalized feeds, reference data, and standard connectivity everywhere the holding horizon can absorb a few milliseconds without the edge decaying. A crypto-native market maker running cross-venue quoting faces the identical decision on a different venue list: build raw order-book ingestion for the two or three venues where the edge is latency-sensitive, license normalized data for the rest.

Here is what that looks like in practice, as a representative case rather than a specific client’s audited numbers. Consider a fund weighing whether to build its own multi-venue feed normalizer. The build path costs roughly two engineer-quarters and a mid-six-figure sum in salary and infrastructure before the first tick is normalized, and it delivers a component upstream of the book, not the signal. The license path costs a fraction of that in vendor fees and frees those two engineer-quarters for the strategy. The return shows up as engineering time redirected to the one component that decides whether the strategy has an edge; the tick-to-trade number barely moves, because a normalized feed handler was never the part of the critical path where the edge lived. I hold specific client figures under NDA, but the shape of that tradeoff, engineer-time freed against latency actually gained, repeats across nearly every build I review.

Hasbrouck and Saar (2013), in the Journal of Financial Markets, found increased low-latency activity associated with narrower spreads, more displayed depth, and lower short-term volatility: the infrastructure investment has a measurable market-quality return, not just a private one, and it accrues only to a build disciplined enough to know where its own edge sits.

The Instrument-Before-Spend Sequence

This is the checklist version of everything above, the sequence I hold every build to before I sign off on infrastructure spend, whether the build is new or the one already running.

1. Instrument first. Deploy hardware timestamps at the network card, or at the furthest point you control on the wire, before any speed-related spend. No baseline, no spend.
2. Establish the two measurements. Wire-to-wire tick-to-trade at p99 and p99.9, and adverse-selection markout at the actual holding horizon. Both, not one.
3. Sequence the critical path. Feed handler, then order book, then signal, then gateway, each proven against the prior stage’s real output before the next stage starts.
4. Decide the colo/cloud split component by component. Only the critical path goes to colo, or to the exchange’s own low-latency access tier. Research and backtesting stay in cloud.
5. Run the build-vs-license test on every component. Ask whether this specific piece is where the edge lives. If not, license it and reallocate the engineering hours.
6. Keep the latency budget honest on an ongoing basis. Re-measure after every material change to the critical path, and treat any p99.9 regression as a production incident, not a rounding error.

A CTO who can walk a board or an allocator through this sequence, with actual numbers at each step, is holding a build to a standard most competitors are not.

Conclusion: The Capital Call That Isn’t Mine

Architecture decides whether the $50M builds a machine or burns it, and the same holds for the next $5M spent modernizing a stack that already exists. Every failure mode here is a sequencing decision, made or missed before the first dollar clears. Holding a build to that discipline, instrumented, sequenced, honestly split between build and license, is the work I get retained for, and in practice it means sitting across from the desk’s own engineers as the review runs, not writing a report and disappearing.

Here is the standard, stated as a test rather than a promise: if your build cannot prove its tick-to-trade at p99.9 with hardware timestamps at the source, you have an unmeasured system, with no way yet to tell whether it is also a fast one.

What I have not closed, and will not pretend to, is the question one level above architecture. Whether a strategy deserves a $50M machine in the first place is a capital-allocation call that lands on the allocator’s desk before it reaches mine as an engineering review. No amount of disciplined sequencing on my side answers it for you.

This article was originally shared as a LinkedIn post.

Ariel Silahian has 20+ years of experience in electronic trading systems architecture, advising CTOs and Heads of Trading on infrastructure sequencing, latency measurement, and build-vs-license decisions across electronic trading venue classes, from TradFi HFT to crypto-native quant funds. For funds that want this sequencing discipline applied to their own build plan before the spend happens, whether that build is new or the one already running, the Discovery Assessment is structured around exactly this kind of review.