Original post by Aiden Pestell
Hi tech savvies. My name is Aiden and I have spent the last 4 years working as a Java recruiter within the Front Office market covering mostly Hong Kong & Sydney. During these years I have found a passion for a market within Java which is low latency Java and more specifically the sequencer architecture.
From my younger years I was a 100-meter Warwickshire champion where I’ve always enjoyed high speeds, so when I started learning about how trading systems worked, low latency was a thing that -ironically or maybe naturally- caught my attention.
Through networking with the engineers and heads of trading, I have developed this knowledge in low latency and more specifically how sequencer works. This has enabled me to connect with the fastest Java developers in the world and engage with Investment banks.
I’ve seen firms who are implementing sequencer getting less than 10 microsecond on their engine with over 2 million transactions per second.
This blog will go into why we should use Java instead of C++ for low latency, how sequencer was born, and who the key people were in growing sequencer out. Of course, it couldn’t end with my personal opinion on that and information I gathered during my years of work.
This blog is ideally aimed for IT professionals and executives who are working on high throughput trading systems and passionate ones that are starting or wish to implement this into their trading systems.
Let’s start!
The first step before even talking sequencer is the obvious: why would you even use Java for low latency?
This is typically the first headache for any architect or head of trading. When first deciding which language and framework you should use, most architects would need to consider a few things whilst looking at performance. Of these it would be latency (time to market on a trade), throughput (how many orders the system can handle) without sacrificing one or the other, and the manpower to build it. The obvious choice to many -when looking at latency and throughput- would be to code in C++, as the code is closer to the hardware and thus better for performance, compared to Java.
Java typically runs on top of the hardware by using a Java Virtual Machine (JVM) which normally can cause a lot of memory issues, resulting in a slowing down in the performance.
To combat this, Java guru Martin Thompson, the engineer who built LMAX Disruptor, believed in Jackie Steward (racing car driver) who coined the term “Mechanical Sympathy”.
“You don’t have to be an engineer to be a racing driver, but you do have to have to have mechanical sympathy”.
This is where an engineer needs to apply an understanding of the hardware to the creation of the software. By understanding the mechanics of the machine an engineer can get the best performance of the system.
By coding Java with the correct data structures, memory techniques, Network decisions and the right architectural design, teams can get close or equal to C++ performance levels. Choosing a User Datagram Protocol (UDP) and Multicasting allows the sequencer to broadcast data to multiple receivers in the most efficient way possible. This then allows the network to speed up the latency on a single-threaded system, making architecture clean and allowing all the nodes on the system to receive the exact same set of messages.
Another reason why Java is more popular to use in the banks is that there is more manpower and therefore it’s easier for the team to hire engineers.
C++ is still a very specialised portion of the market, and this usually causes headaches for any hiring manager in terms of finding the right talent, retaining them, and not paying through the roof to even have them.
As most of the banks use Java, hiring in this language makes complete sense when looking at all the factors needed for building a successful low latency trading platform.
The ideal way would be to do code in Java and teach developers to have mechanical sympathy and learn techniques mentioned before that will allow the optimal performance when building the system.
This brings me onto want to write about the architectural design that was used for the first electronic trading system ever built without human input and an architecture that I am passionate about: the Island ECN Sequencer architecture.
Sequencer is a single threaded UDP multicast middleware messaging bus that essentially multicasts all messages into a sequence of numbers. The architecture is fully deterministic, meaning that every node sees all messages in the same order and all events can be replayed on a single stream producing the exact same state through finite-state machines. It is seen as a simple design yet still today is the fastest architecture that firms go for even if it’s over 20 years old.
The next part will go into detail on its origins and how the sequencer has gone from Broad Street to all the way around the world with top crypto exchanges, investment banks, market makers (the largest, fastest and most profitable ones) and buy-side firms for their entire global systems.
Levine’s Sequencer & Island ECN Influence
Let’s go back to the early 90s in Wall Street where former college dropout Josh Levine worked at Datek Securities (Now known as TD Ameritrade).
Datek’s traders became successful and popular for their practice -created by Levine- of bypassing market makers who were still updated their quotes by hand at NASDAQ. This of course benefitted Datek and its traders, until NASDAQ found out and penalised the company.
Patterson in Darkpools mentions:
“The Nasdaq market makers, and the waves of new rules and obstacles that NASD kept ramming out, were a constant source of anxiety and outrage at Datek. Market makers routinely ignored Datek’s SelectNet orders, backing away from their posted quotes, often leading to missed opportunities or, worse, significant losses as traders, struggling to get out of positions in a fast-moving market, were left flapping in the breeze.”
Levine saw an opportunity and decided to start his own shop officially in 1996, Island ECN. Island was registered on the 6th floor of 50 Broad Street (New York) and this is where the sequencer framework officially started.
Now Levine could essentially bypass NASDAQ market makers by building the sequencer messaging bus that would match all the buy and sell orders and then execute a trade without any human input. Levine had built the engine on FoxPro and ran on DOHS (You can find his FoxPro code here.).
By building a system that could do this, Levine became the most important figure of modern day markets.
Being a guy who didn’t want to be famous and tried to keep himself private to the attention, Levine locked himself in his office and dedicated his time in creating a fast, simple and scalable system. Basically, something that could handle anything anyone threw at it. By the end of the 1990s Island ECN were taking 10% of all stocks traded on the NASDAQ & they were executing 257,140 daily trades (Compared to 28,446 IN 1997). This was a huge problem for NASDAQ as they were seeing massive volume of trades from this new competitor, who were reporting the trades back to NASDAQ. However, this caused a drain on NASDAQ’S liquidity as they were not benefiting from these trades.
Over these years, Island had built a great engineering team around Levine which consisted of William Stirling, Mike Blum, Jonathan Ross and arguably the most important engineer being Brian Nigito. (EX Tradescape). Levine saw him as his protégé and would say that Nigito was a better programmer than him.
By the early 2000s Island couldn’t keep up with the demands, which resulted In 2002 Instinet acquiring Island for $508 million. This resulted in Levine leaving as he didn’t want to join a rival firm that he always despised.
In 2003, eventually, the Island team split up and Nigito left to lead the 36 man charge at Instinet (Inet) -working closely with Constantine Sokoloff, Nigito reengineered Levine’s FoxPro Engine into Java. At the time Jonathan Ross was the CTO of INET, and it’s important mentioning as he was a key person in driving sequencer at NASDAQ and CHI-X with Nigito. To this day Nasdaq’s INET ECN still uses the sequencer architecture handling billions of messages per day.
On the other side, Stirling and Blum left and joined UBS, which in 2004 became the first Investment Bank to implement the Island architecture.
They built UBS’ global foundations for Equities systems in Stanford, which by the way are still used today (and it’s part of their core low latency technology).
After a few years, in 2009, they both departed to grow out sequencer architecture elsewhere and became respectively the CTO of Teza and Getco (KCG), here Blum rejoined his ex-Island colleague Nigito.
Blums influence hadn’t stopped there, as in 2017 he became a partner at Goldman Sachs, where he rejoined fellow Islander Steve Lui to lead the Atlas sequencer project on the Electronic Trading team. Goldman invested millions of US dollars in order to improve the equities business and Blum mentioned “One goal was to improve the firm’s performance on trade orders that last a microsecond (a millionth of a second) to about 30 seconds, shorter-lived orders that the bank may have struggled with in the past. In some instances, trades that took hundreds of milliseconds will happen faster than 100 microseconds”.
With Goldman Sachs becoming the 2nd Investment Bank to implement Island’s architecture globally, Crypto firms and other Investment banks have now started to follow suit especially in APAC. With Crypto area now and having a large number of developers I’ve seen an influx of non-islanders growing this out which makes me think that this architecture -which is so old- will continue to spread and be learnt throughout the next years to come.
Sequencer still not the go to
However, the sequencer messaging bus is still niche within the fintech world as a lot of trading platforms are not using it. The main reason is that it’s hard to get the architecture done right as you need a team of sequencer professionals to build it. These are hard to find and mean that you are entering a niche market which will be costly and will take a long time if built in-house. It takes a long time to mature such a complex system, full of moving parts and corner cases.
Secondly, a lot of managers would rather tackle throughput rather than latency and feel that a multithreaded trading system would be better suited to handle more high touch products and thus won’t want to go down the single threaded sequencer route. Thompson’s LMAX disruptor is a popular choice for multithreaded trading systems, but this can bring multithreading pitfalls so common to developers and eliminate determinism. The single-threaded approach provides more robust and bug-free code.
Furthermore, to even take on this investment would mean starting from scratch, which a bank would be reluctant to do as it will cost them millions. For a firm who has most of the market share you can imagine why they would be reluctant to change it. If it’s not broken why fix it?
To slightly reduce the risk on investments with time to market, a lot of banks and crypto firms are using licensed low latency vendors that can get their networking and engines running within months. There are old ones that banks have been using, such as: Tibco and Solace, which are slow but popular with the banks. These are generic messaging buses and not a reliable multicast UDP networking one.
The Future
To conclude, it’s apparent that the sequencer growth into the trading world has been reliant on the Island ECN guys & has seen a few ripples of further growth over the past 20 years. Looking into today’s market, with the rise of crypto and more banks implementing sequencer since GS, I believe it’s going to grow twofold with years to come.
Levine created something so simple and fast and it’s only now that trading firms are starting to go with it. If you see how well the investment has paid off for these firms that can handle millions of orders per second in sub micro java then you’d also do the investment into hiring a bigger pool of active Java programmers, firms can have the manpower to get started in the market.
Furthermore, there are a few Sequencer vendors that I can introduce you to, and this will then get you up and running within 6 months and thus not needing to hire the core guys for it. Moreover, banks investment goes in cycles but if you are currently at the top and decide not to invest, you will only be left behind the competition. The potential is endless from Levine’s sequencer, It started at 60th Broad Street and I believe that sequencer can go further than trading and one day we will see it in space.
