A month ago I wrote a post-mortem about a failover test that stopped the wrong node: preview.8 shipped election code that nothing in the server called. I ended it with a promise. The article about elections would wait until elections actually run.

In 0.1.0-preview.9 they run. Stop the leader of an RF=3 group and the other two elect a new one, the client keeps working, and the leader-stop tests kill the node that matters. This is that article.

Part 1 covered the replicated write path: synchronous mutations, majority commit, write-ahead idempotency intent, and the honest CommitOutcomeUnknown answer when the owner dies mid-commit. All of that assumes something it never proves: that exactly one node speaks for a replica group at a time.

Without that guarantee, two survivors of a partition can both believe they lead, accept different writes for the same key, and return success for histories that can never be reconciled. Majority commit on top of two leaders is just durable disagreement.

Squirix is still an experimental preview, not a production-ready cache. Its APIs, wire format, and storage formats may change during 0.x. This article describes the current design, not a compatibility promise.

Opt-in, and only for three or more

Automatic failover is off by default. Two settings under Squirix:Cluster turn it on, and they come together:

{ "Squirix": { "Cluster": { "ReplicaCount": 3, "ReplicationEnabled": true, "AutomaticFailoverEnabled": true, "QuorumReadsEnabled": true } } }

Validation refuses one without the other, and refuses both for groups of one or two replicas. Every node needs the same values, and they are set on empty data directories: the switches are part of the topology fingerprint.

With failover off, nothing changes from preview.8: the owner of each key leads its group in term one, reads are local, and losing the owner makes its keys Unavailable until it comes back. Everything below is about the switched-on mode.

The failure mode to avoid

Imagine five nodes split 3–2 by a network cut. The minority side bumps its own term number and declares itself leader. It accepts a write for key K. Meanwhile the majority side elects a real leader and accepts a different write for K. The partition heals. Which write won? Both clients got success, and the log has a fork nothing can merge.

The contract that prevents it borrows the oldest Raft lesson:

no node leads on its own authority | v a candidate wins only with a majority of the configured static membership | v votes come only from members whose log can vote, but that never shrinks the quorum | v a leader that stops hearing from a majority steps down

A lone node can inflate any number it likes. Without a majority behind it, the number means nothing.

When an election starts

A follower campaigns when it has heard neither from a leader nor granted a vote for one election timeout: one second plus up to one second of seeded jitter. The leader sends a heartbeat to every idle follower every 100 ms, so a live leader keeps everyone quiet.

A campaign starts with a pre-vote for the next term, which changes no term anywhere. A member refuses it with leader-contact if it heard from a live leader within the last election timeout. That one rule matters more than it looks: a node cut off from the leader alone would otherwise keep raising the term and depose a perfectly healthy leader every time it reconnects. With the rule, it campaigns in vain until it can hear the leader again.

Only with a majority of pre-votes does the candidate persist the new term and its own vote, then ask for real votes. Each vote request has a 250 ms timeout, and a round ends as soon as its outcome is decided, so a dead member does not slow a round that already has its majority. That early end made promotion with a silent old leader much faster in this release: about 0.3 seconds instead of 2.3.

Term one always belongs to the owner of a group, which leads it without votes after the group's first start. The first election is for term two.

Terms that survive the process

Every replica group stores a durable current_term and at most one voted_for per term, in both slots of its metadata file. A vote is persisted before it is sent — the same journal-first instinct as preview.6, applied to elections. A node that crashes after voting remembers, after restart, who it promised the term to. Double votes across restarts are not a matter of discipline; the disk forbids them.

Voting adds a second gate: log freshness. The voter compares the candidate's last log term, then its last log index, with its own, and refuses a stale candidate with stale-log. A new leader that never saw a committed entry cannot win, so it cannot overwrite that entry — the classic "elected but amnesiac" failure.

candidate (lastTerm=4, lastIndex=120) --> voter (lastTerm=5, ...) | +--> refuse: stale-log

Some members cannot vote at all. A copy that came back on an empty disk is unseeded: it grants no vote and takes no entries until the group repairs it. A copy whose log is not ready refuses every ballot. A copy waiting for a state transfer still votes, because its log is real, but it can never stand for election.

Authority must be earned, then kept

Winning an election is a moment. Serving traffic is a stream.

A winner does not serve anything yet. It first appends a leader-noop record in its own term and waits until a majority commits it. Until then every write is refused before anything is appended, with Unavailable and "nothing was read or written". Only that current-term commit lets the leader treat older entries as committed. This is Raft's leader-completeness rule stated as a startup step, and it closes the window where a new leader serves history it does not yet own.

Keeping authority is the other half. A leader steps down once fewer than a majority, itself included, answered within one election timeout, and at once when any reply carries a higher term. A deposed leader's messages carry a stale term, and followers refuse them with stale-term instead of merging them.

Some leaders are alive but useless: one that cannot get its own term committed, one whose apply loop is stuck, one whose own log appends keep failing. Each of those gives the group up after 30 seconds so another member can take it. A stopping leader hands the group over before it drains, instead of letting it wait out an election timeout.

Reads that refuse to guess

With quorum reads on, a read on the leader goes through five steps:

1. check authority in the current term 2. read_index = commit_index 3. confirm with a majority in that term (requests sent after step 2) 4. wait until applied_index >= read_index 5. return the value

A minority side or a former leader cannot pass step 1 or step 3. It answers Unavailable (read_quorum_unconfirmed, read_index_unapplied) or a stale-term refusal — never an old value presented as current. There are no time-based lease reads: authority without a quorum is never enough, even when clocks agree.

Finding the leader

A client should not have to know who leads. Any node accepts a single-key call, including a node outside the key's group, and forwards it to the leader it knows. If the answer is stale-owner or stale-term, it reroutes once to the leader the answer names. All the leader waits of one call share a three-second attempt budget, and the client's retries share one 15-second deadline. Without a leader the call fails with Unavailable and nothing is written, so a retry with the same operation_id is safe.

client --> any node --> leader it knows (term 7, current is 9) | +--> stale-term: one reroute to the named leader, then fail visibly

Bounded matters. "Try the next node forever" turns every leadership change into a thundering herd from the clients.

Every replication RPC also carries the term, the topology fingerprint and the configuration generation in its header. A message from another world — wrong group, older configuration, a node below the minimum cluster version — is refused before it touches the log.

Why RF=2 stays a mirror

The same majority rule that saves RF>=3 condemns RF=2 to honesty.

With two members the majority is two. Lose either node and the survivor holds one vote out of two. It cannot elect itself, cannot commit, cannot promote. That is not a missing feature; it is the protocol refusing to invent authority it does not have. Validation does not even let you switch failover on for RF=2.

RF=2, both alive: synchronous mirror, every write on both nodes RF=2, one lost: writes stop, committed data stays readable locally, no promotion RF>=3, one lost: with failover on, the other two elect a leader and keep serving

Two-node majority systems that promote on peer loss are single-node systems with extra steps — and extra data loss.

Coming back

A node that returns after an outage catches up from the leader's log. If the leader's log no longer holds what it needs — it was compacted, or the node came back on an empty disk — the leader sends a state transfer: its snapshot plus every cache entry of the group. A wiped copy also forgot whom it voted for, so before repairing it the leader confirms with a majority of the other members that no newer term exists. That check makes the forgotten vote harmless.

A returning node does not take the group back. It stays a follower: its peers refuse its pre-vote while they hear the current leader, and nobody votes for a stale log. Membership is not readiness to lead.

A model that is allowed to say no

Before any election code landed, I froze the protocol in an ADR with an executable C# model (src/squirix.protocol-model, never shipped, never referenced by product code). It has immutable state, pure transitions, a deterministic BFS explorer, and invariants that report minimal counterexample traces.

The explorer covers elections, commit, and quorum reads for RF=2 through RF=5, with crash points before and after every durable write of term, vote, log, and commit index, and with loss, duplication, reorder, and one-way isolation of a node. Newer profiles add disk loss and repair. A profile that hits its state cap without a violation exits differently from a clean fixed point, so residual risk stays visible instead of silently passing.

Just as important are the broken variants that must produce counterexamples:

  • grant a second vote in the same term;
  • compare log length before the last term when voting;
  • commit an old-term entry without a current-term commit;
  • serve a read without a majority confirm, or before apply catches up;
  • repair a wiped copy without the term ceiling.

If the explorer ever stops finding those violations, the harness is broken, not the protocol sound. Conformance tests run real product components and ask the model whether it can reach the same trace.

The post-mortem was about the gap between this model and the server. The model says the protocol is safe; it never said the server runs it. What closes the gap now are the E2E tests: an RF=3 and an RF=5 cluster whose leader is stopped, a leader isolated by a partition, a five-node 2/3 split with the leader in the minority, a wiped follower repaired by state transfer, and a former leader that restarts and stays a follower. The tests that stop a node assert that it was the leader.

What preview.9 does not promise

Membership is static: no dynamic membership, rebalancing, or joint consensus. Election timings are fixed. The model covers single-node isolation; the five-node split is covered by a test, not by the model, and richer partitions remain residual risk. A group that lost the disks of a majority of its members at once is not repaired.

Closing thought

Commit protocols ask: do enough nodes hold this write? Election protocols ask the harder question first: who is allowed to ask?

The hard part is refusing authority precisely:

  • no majority, no leadership — even if the node feels healthy;
  • stale log, no votes — even with supporters;
  • no current-term commit, no serving old history;
  • no quorum confirm, no current read;
  • a live leader, no pre-vote for anyone else.

Preview.9 makes each refusal durable, fenced, and observable — and, this time, switched on. The final article in this series covers what operators see: the opt-in, metrics, readiness, and an RF=3 compose example you can run.

If you build leader election or quorum reads in .NET, comments are welcome on where you draw the same lines — especially vote persistence, pre-vote, and rerouting budgets.

Links

Squirix docs:

In this series:

Try it:

dotnet add package squirix --version 0.1.0-preview.9 dotnet add package squirix.server --version 0.1.0-preview.9 dotnet tool install --global squirix.server.tool --version 0.1.0-preview.9