Capital markets resilience and recovery with Aeron Sequencer: A property you can prove, not a story you tellThis is the third in the Aeron Sequencer series that began with Capital markets infrastructure: The end of the trade-off between performance and resilience, and Trading system development with AI: cost moves from writing code to trusting it. It takes one of the four properties the opening post argued a trading system has to hold at once, resilience, and explores how the architectural design of Aeron Sequencer means you can speak about resilience and recovery using demonstrated behaviour rather than referring to error prone human processes and run books.
This post is for you if you run mission-critical systems in Capital Markets where every second of uptime is critical to the running of your business and the footprint of your technical estate means your resilience and recovery strategies vary across services and, in many cases, rely heavily on manual processes and runbooks.
When looking at resilience, one way to think about the availability of a system is to look at the following four layers:
We can ask the same question of each: When did we last exercise the resilience of the layer, during trading hours, with the book full, without anyone being warned?
At one end of the spectrum, the answer is factual: it works, it is exercised constantly, here are the dates. At the other, it is a document of manual steps towards an intended outcome, and a belief that they would work. Most estates run the whole spectrum, and few can say precisely where fact becomes belief.
Two decades of reasonable design choices have left most firms able to describe their resilience accurately and prove it only in part. Description used to be enough. It is not what risk committees, regulators and participant due diligence reviews ask for today.
This post is the first of two on resilience and recovering from failure. In this post, we look at the design of Aeron Sequencer cluster failover, which means we can demonstrate resilience to, and recovery from, localised failure as a property of the design rather than a story we tell for each layer of the deployment architecture. The second post looks at how we handle failure when an entire region or data centre goes offline.
On-premises, failures are physical and partial. A network card fails. A disk fails or fills. A top-of-rack switch freezes on a firmware upgrade. The machine may keep running with partial service.
On cloud, failures arrive as things moving underneath the workload. Instances are retired and restarted. Network infrastructure is replaced. Whole systems migrate onto other machines, sometimes on the provider’s schedule rather than the firm’s. Components may fail, or something simply disappears and reappears somewhere else.
These are not the same problem. A partial failure can be more complex than a clean one; a machine that is up and reachable and no longer doing its job may not trip the checks that a machine which has simply gone would. Cloud failures are often announced in advance; instance churn is routine rather than exceptional.
What they have in common is something holding state has stopped contributing, the rest of the estate is still trading, and the replacement has to arrive at the right state. The rest of this post walks the layers where one mechanism answers that, in both worlds.
Process death is a common failure mode, and the one most thoroughly solved: an orchestrator restarts the process, or a hot standby instance is already waiting.
Restarting a process and recovering its state are different problems. The process may be alive but not up to date with the latest events in the system.
Picture an order router running in a container with a fixed memory limit. During a volatile open, a burst of venue acknowledgements arrives faster than it can work through them, the queue in memory grows past the limit, and the container is killed. The orchestrator restarts it within seconds, and it replays the venue acknowledgements it missed, taking a period of time to clear the backlog of messages. The still running position service consumed those same acknowledgements while the router was down and has already moved those orders to filled. For that period, one service shows dozens of orders working that the other shows filled, and the desk cuts limits until they agree. Depending on the architecture and reconciliation needs, that period can be seconds, minutes or hours.
The difficulty is not that recovery code does not work. But, it is written once per service, by whoever built that service, and it runs only when something has already gone wrong. The least exercised code in an estate is the code that runs on the worst day.
Aeron Sequencer moves that work out of the application. A service rebuilds by applying the same deterministic logic to the same ordered inputs that built its state originally, starting from a periodic checkpoint rather than from when the system was last re-started. The replay of messages exercises the same code as normal message processing; doing the same thing to the same inputs in the same order. There is no separate recovery mode to write or get wrong. A service consumes messages the same way on its worst day as it does every second of every day. The pattern for recovery is baked into Aeron Sequencer and can be used on any project or service across an integrated enterprise, so we don’t end up with a large jumble of poorly maintained and rarely exercised recovery patterns across a large and complex estate.
A common approach to mitigating host or rack failure is to use a primary and a standby instance: one instance serves, the standby waits to take over. As with the approach to application failure above, this tends to be written on a per-service basis, by whoever built that service, and the code that promotes the standby runs only when something has already gone wrong. Across an estate, that becomes a jumble of failover and recovery code spread over a multitude of services and a multitude of HA pairs, and the question is: when was each of them last exercised in anger?
A real world scenario: A pre-trade risk engine runs as a hot pair (fig 1 below): a primary that serves, and a standby on another host. The primary publishes each approval and fill to the standby over multicast, so the standby’s view of positions should match. Each update carries a sequence number, and if the standby misses one it asks the primary to send it again. If the primary stops responding for three seconds, the standby takes over.
Mid-afternoon, in a busy second, the primary approves a burst of orders and they fill at the venue. The updates arrive at the standby faster than it can take them in, its receive buffer overflows, and it drops them. It spots the gap and asks the primary to resend. Before the resend arrives, the primary’s host drops off the network.
Three seconds later the standby promotes. The heartbeat triggers the promotion, and nothing in that logic checks for an open gap. The standby’s view of positions is missing the burst of fills, so it believes the account has room it no longer has. The next order arrives, the standby approves it, and the account is over its limit. The only trace is a gap warning in the standby’s log.
The standby stayed up, but it was wrong, and there was nothing it could have done about it. The updates it needed existed only on the host that had gone. Monitoring might flag the gap within seconds, but by then the order is at the venue, and the fills can only be pieced back together from other systems at reconciliation. A more careful standby could refuse to promote with an open gap, and then the risk engine is down instead. The pair can be unavailable or wrong. It cannot be both up and correct.
Now let’s look at the same risk engine, running as two replicas consuming messages from Aeron Sequencer (fig 2 below), and the same network card failure at the same moment. Aeron Sequencer keeps one ordered record of every instruction it accepts, and an instruction only counts as accepted once that record has been copied to a majority of the cluster nodes.
Node one is still running, but it can no longer reach the other two, so it cannot count itself into a majority and it stops acknowledging. Nodes two and three confirm they can reach each other, count two nodes of three, and carry on. Only one group can ever be more than half, so a group that counts itself into the majority knows no other group is also running, and nobody has to adjudicate. The risk engine service simply continues reading messages from the cluster: a consistent view of the world.
Every order the risk engine approved was, by definition, accepted by a majority of the cluster nodes before the approval left. Losing node one cannot have taken any of them with it, because accepted means at least two nodes hold it. Nodes two and three have every approval the engine made, in the order it made them, so their position pictures are identical to each other and identical to what node one held. The next order for that account is checked against a picture that includes every fill. It is rejected, correctly.
The service stayed up. It lost nothing, and that is a property of the design rather than a hope. It was consistent because the survivors were never behind. This scenario could equally have maintained the consistency, uptime, and ordering guarantee for a matching engine, an OMS, an EMS, a SOR, a netting service, or a clearing service.
Rack failure is the same mechanism with one more placement decision. Put the nodes in different racks and the failover that covered a lost host now covers a lost rack with the same guarantee, and with no new configuration to get wrong. On cloud the provider hides racks, so the equivalent decision is which placement group the instances sit in. Either way it is one decision, checked once, for every service that reads the record, rather than one per service, host, rack or placement group written separately by whoever owned it.
With Aeron Sequencer this layer gets exercised constantly. Every software patch, every instance replacement, every failed disk takes a node out of the group, and each one is the failover path running in production during trading hours. The code that takes over is not the least exercised code in the estate. It is among the most exercised, which is why the answer here is a date from this month rather than a description of what should happen.
Picture a cooling failure taking one hall of a metro pair offline mid-afternoon. The database pair switches over cleanly; it was built for this. The OMS application has to start in the second hall and rebuild, which takes minutes. The risk service, in the surviving hall, never stopped. It is still authorising orders against a position picture that stopped advancing the moment the OMS failed over to the backup. All services are running and reporting healthy, but the services do not agree.
An estate with a metro pair will have had considerable engineering time spent on it. Two things are worth asking.
What does synchronous data replication do when the link degrades rather than fails? The system has two choices: stall until the secondary acknowledges, or continue without waiting and lose the guarantee until the link recovers. In the common configuration it continues, which makes the guarantee conditional on link health in a way that is not visible until somebody reads an alert.
When a hall goes, every service in it fails over at once. What coordinates them? Each has its own HA design. Some are active-active, where losing an instance is invisible. Some have a hot standby already reading the same feed. Some are provisioned but not fed, waiting for somebody to start them and warm them. Some are a set of provisioning scripts waiting to be run on the day. At the host layer one of these fires. Here all of them fire together, each designed separately, so they do not come back to the same point. One service is live and current, another is minutes behind and catching up, a third has not started. Nothing reports when they are aligned, so somebody watches and decides when it is safe to resume.
On cloud, Aeron Sequencer node failover covers this layer the same way it covered the last two. Place the cluster nodes in three availability zones and losing one is the same event as losing a host: the survivors are a majority, they carry on, and the applications reading the record in the surviving zones are already current. There is no feed-and-warm sequence; nothing was waiting to be fed. The write waits for the quicker of the other two nodes rather than one designated partner, so a degraded link between zones never forces the choice between stalling and quietly dropping the guarantee.
On-premises the constraint is different; most firms have two data centres, not three. Three nodes can be placed across racks inside one of them, but a majority cannot be placed across two sites, because there is no third site to put the deciding node in. Losing a whole data centre or cloud region in that topology is not something node failover can absorb on its own. It is a disaster recovery event, the subject of part 2 of this blog, coming next.
This table shows the stateful services that today get their resilience from a bespoke set of resilience and recovery strategies and run books and how Aeron Sequencer simplifies this.
| Failure domain | Typical mechanism | Why this means a human process is necessary | Aeron Sequencer node failover: a repeatable automated process |
|---|---|---|---|
| Application | Orchestrator restart, or a hot standby waiting on the bus; state rebuilt from a database or replayed from a journal. | The restart is instant, the rebuild is not. The code that does the rebuilding is written per service and runs only when something has already gone wrong. | Recovery belongs to the platform. The replacement replays using the same code path it uses for every message. |
| Host & rack | A primary and standby pair per stateful service, hot, warm or cold. | The pair cannot settle alone whether the primary is dead or merely unreachable, and there is no means of guaranteeing the standby is current. | The survivors count themselves into a majority without a witness, and are current by design. Every patch and every retired instance exercises it. |
| Zone | Synchronous replication across a metro pair; a standby application tier in the second hall. | Every service in the hall fails over at once, each on its own design, and somebody decides when they have the same view. | The same node failover, nodes across three zones. The surviving zones are already current and have a consistent view of the world. |
Each of those mechanisms was the right tool for the layer it protects, chosen by the team that owned that layer, at the time that system arrived. What the sum lacks is an approach with correctness baked into the design, so that resilience and recovery are properties that behave consistently and predictably rather than differently at every layer for every project across your estate. The open question is who can say what the firm’s picture looks like after a failure that crosses two failure mechanisms, and whether those two have ever been tested on the same day, by the same people, against the same definition of recovered.
At every layer below the region, Aeron Sequencer’s single ordered sequence gives the same answer, because it is the same mechanism placed a little wider each time, and it is exercised every time a host is restarted or patched.
Go back to the diagram. At each layer, the question is when it last worked during trading hours, with the book full, without warning. When resilience and recovery is baked in by design that answer is a provable date and can be put in front of a regulator. Where the answer describes how it should work, with people, process and steps to work through, it is a story.
For every layer node failover covers, that date should be recent and unremarkable: a host was patched, a node went with it, and nobody on the desk noticed. If it is not, the gap is worth knowing about before somebody asks for evidence of it.
For anyone who has asked those questions of their own estate and found the layer where the evidence stops, that is the conversation we would like to have.
Ralph is a Strategy Executive at Adaptive with 20+ years in capital markets tech, driving exchange, matching engine, and Aeron-based product strategy for global trading venues and fintech leaders.
Learn how leading firms can reduce operational risk, simplify distributed architectures, enable active-active deployment, and build resilient trading platforms that scale.
Martin Thompson shares how a replicated state machine design can replace much of the complexity that accumulates in traditional microservice architectures.
See how leaders in exchanges, banks, and trading firms are modernizing their infrastructure.
In this webinar, we tackle one of the toughest architectural mandates in capital markets: uninterrupted 24/7 operations.
LinkedIn profile