Eventual consistency: When your systems disagree on an order status, the firm acts on the wrong pictureThis is the first in our blog series on the challenges of building trading systems today. The anchor post, Capital markets infrastructure: The end of the trade-off between performance and resilience, sets out the four things capital markets technology leaders are now asked to deliver at the same time: consistency, performance, resilience, and maintainability, and why the old answers no longer stretch to cover all four.
This post takes the first quality attribute, consistency, and peels back the layers:
Each post in the series stands on its own and can be read in any order. This post assumes only that you have built trading systems out of services that talk to each other.
Let’s start with the pervasive architecture in Capital Markets. A position service holds the firm’s working positions. A risk service consumes those positions to make pre-trade decisions. A client blotter shows the user what is live. The three communicate over a message bus, and each owns its own state. This is the order-handling, position, and risk tier a sell-side or buy-side technology leader has spent a career building and operating, and on its own terms it works.
That pattern is not unique to the buy-side or the sell-side. It is what happens wherever services are distributed and kept in step by propagating events between them, each holding its own copy of the state. Those copies can disagree about the same thing at the same instant. The same shape runs inside an exchange or a Central Counterparty Clearing House (CCP), downstream of the matching engine, where post-trade, clearing and margin, market surveillance, and regulatory reporting each consume the order flow on their own path and each build their own view of what happened.
The promise of this architecture is independence. Each service owns its slice. The message bus carries events between them. Eventual consistency is the pattern the firm chooses, and the price it pays, for independence between services and teams.
Let’s look at a common scenario, in the sunny-day case where every service is healthy and nothing has actually broken: the position service, the risk service, and the client blotter hold different views of the same order, even if only for milliseconds, and the firm acts on whichever view it happened to read first.
On the participant side, pre-trade risk passes a child order because positions on the risk side are one fill behind the position service. The firm breaches a limit it believed it was comfortably inside. Inside a venue, downstream of the matching engine, market surveillance, and the regulatory-reporting feed consume the same execution flow on separate paths and assemble it into slightly different orderings; when a regulator later asks for the sequence of events in a one-second window, piecing together what the two systems saw takes days and the answer is two different sequences – nobody can say which is authoritative without rebuilding the input sequence by hand.
None of that required a fault. The harder truth is that the gap does not stay at milliseconds. It widens the moment the system is under load, and no service has to crash for the firm to be badly wrong. It only has to fall behind, or be briefly cut off.
Then the cost arrives: when a regulator asks what the firm believed at 14:32:07, the answer is reconstructed from disparate logs after the fact, and reconstruction is where audit findings come from. The reputational, regulatory, and operational costs compound together. Eventual consistency is not an auditable or defensible guarantee.
And all of this is before a single service crashes and has to recover. At this point the question becomes whether it comes back holding a view consistent with the services that never went down. That is a property the firm can either prove or cannot, and it is the subject of following posts in this series.
The reasonable response is to ask why the tools already in the stack do not solve this. Let’s look at each in turn, because each gives the firm a real property, just not the one needed here.
A database in the fast path gives strong consistency to whoever writes to it, and eventual consistency to everyone downstream reading from it. The position service may be authoritative, but every other service is reading a view that is already behind, and a database in the hot path is a performance problem as much as a propagation one. The database is not wrong; it is simply not the architectural choice that makes the other services agree.
A distributed cache brings its own ordering and deduplication problems. Two consumers can see two different orderings of the same writes. The cache hides the inconsistency from your latency profiles, not from your audit trail.
A message broker typically only preserves ordering within a single stream (or topic), and loses it across them. A position update carried on one stream and a risk decision on another can be consumed in either order, and the firm has no way to assert which one actually came first.
The point is not that these tools are wrong. They are correct for what they do. The point is that none of them give the firm a single ordered view of what happened, when, across services, and that single ordered view is the property the firm is really paying for in the middle of a reconciliation outage.
The computer-science answer has two halves, and both matter. The first is total ordering: every component reads the same messages in the same order. The second is single ownership of state: one component is authoritative for a given piece of state, so peers are never reconciling competing copies. Put the two together, and if every component reads the same messages in the same order from a single source of truth, they cannot disagree about what happened, or when. The problem is structural, so the answer has to be structural too.
That one move closes every failure from the previous section. The sunny-day drift goes away, because there is one sequence rather than several views converging at their own pace. The lagging consumer stops being dangerous in the same way, because a slow service is merely behind on the one agreed sequence, reading the same messages a moment later; it catches up to the same view of state, it never arrives at a different one. Reordering and double-delivery stop being each service’s problem to solve, because order and identity are fixed once, at the source, leaving no private deduplication logic to disagree. And a partition can no longer split one truth into two, because there is a single authoritative sequence instead of two services each extending their own.
This is why monitoring, reconciliation, and careful coding are not enough on their own: they all act after the disagreement is already possible, whereas ordering the messages once, up front, removes the possibility.
The idea is not new. This was mapped out from the 1970s and 1980s by Leslie Lamport, Barbara Liskov, and Ken Birman, and it is the property that infrastructure outside finance, where disagreement is not an option, has run on for decades. In air traffic control, in nuclear control rooms, in ship command and control. What is new, is that the latency and throughput a trading session needs has only recently become viable on commodity infrastructure. The structural answer is to combine multiple input streams into a single ordered sequence before services process them, rather than reconciling their separate views afterwards.
Aeron Sequencer does this through its Sequence; a single, globally-ordered log of messages.
Every message the system handles, an order, a fill, a cancel, a market-data tick, a control message, is appended to it in one total order, and every message on it carries three things:
Every service reads from the same Sequence.
It is worth being precise about what the Sequence is not. It is not a single shared database that every service reads from and writes to, (i.e., the fast-path database problem wearing a different hat). It is a shared, ordered sequence of messages. Each service still builds and owns its own state from that sequence, exactly as it owned its own store before; the difference is that because every service consumes the same messages in the same order, the states they build cannot disagree.
Connect that back to where this blog post started. The position service and the risk service no longer hold separate views of state that drift apart. They read the same messages in the same order from the same source, the client blotter sees that same ordering, and the audit trail is no longer something reconstructed after the fact; it is the same ordered record the system actually ran on. Where a set of microservices gives the firm a weak consistency model, the Sequence gives serializable and linearizable semantics, which is the easiest model for engineers to reason about and the only one a regulator can read off the platform without anyone reconstructing it.
What capital markets firms get follows directly from that property.
✔️ The audit trail reproduces by construction instead of being reconstructed after the fact, which removes a recurring source of regulator findings and turns an incident post-mortem from a multi-day reconciliation exercise into a replay.
✔️ Pre-trade risk can no longer run ahead of, or behind the positions it is checking against, which removes a class of limit breach that no amount of monitoring catches in time: because the breach was already in flight before the monitoring saw it.
✔️ And the question a COO or a regulator asks, what did the firm believe at 14:32:07, has one answer that both of them can read off the same surface, rather than several answers reassembled from logs that were never meant to agree.
Consistency is not something a firm tightens with more monitoring or buys back with one more reconciliation job. Those are defences mounted after the disagreement has already occurred. Order the messages once, at the source, and the disagreement has nowhere to form: the position service, the risk service, and the blotter read the same sequence in the same order, so there is no second version of 14:32:07 to reconcile, no limit checked against a position that has already moved, and no audit trail to rebuild because the record the system ran on is the same record the regulator reads.
Eventual consistency was the price a firm paid for independence between its services and teams. A single ordered Sequence keeps that independence and stops charging the price.
Consistency is the first property a regulator asks about. The second is whether the firm can prove its disaster-recovery story holds, rather than merely describing it. That is the next post in the series.
Ralph is a Strategy Executive at Adaptive with 20+ years in capital markets tech, driving exchange, matching engine, and Aeron-based product strategy for global trading venues and fintech leaders.
Learn how leading firms can reduce operational risk, simplify distributed architectures, enable active-active deployment, and build resilient trading platforms that scale.
This webinar delves into sequencer-style setups and discusses how sequencer architectures support crucial quality attributes.
Martin Thompson shares how a replicated state machine design can replace much of the complexity that accumulates in traditional microservice architectures.
See how leaders in exchanges, banks, and trading firms are modernizing their infrastructure.
LinkedIn profile