The MySQL primary was still running. Applications could reach the database. Yet some write requests were taking too long and timing out.
The problem wasn't a missing primary. Both replicas had stopped replicating, and semi-synchronous replication had become part of the availability problem.
In Part 1, I covered deploying MySQL with MOCO. Part 2 covered backups, MinIO, and a davfs2 failure. This time, let's look at the replication behavior that matters when things go wrong.
The three-instance cluster
MOCO uses GTID-based semi-synchronous replication for its normal three-instance MySQL clusters: one writable primary and two read-only replicas.
Application | v eks-db-1 PRIMARY | binary log events / \ v v eks-db-0 eks-db-2 REPLICA REPLICA \ / replication ACK With asynchronous replication, the primary can acknowledge a committed transaction to the application without waiting for any replica to confirm receipt.
With semi-synchronous replication, the primary waits for the configured number of replicas to acknowledge receiving the transaction's binary-log events. An ACK does not mean the replica has already applied the transaction.
With MOCO's normal three-instance configuration, one replica acknowledgement is required. Losing one replica does not necessarily stop writes; the other can still acknowledge them. Losing both removes that acknowledgement path.
The configuration that changes everything
On the primary, these commands reveal the relevant settings:
SHOW GLOBAL VARIABLES LIKE 'rpl_semi_sync_master%'; SHOW GLOBAL STATUS LIKE 'Rpl_semi_sync_master%'; A configuration captured during the investigation included:
| Setting | Value | Why it matters |
|---|---|---|
rpl_semi_sync_master_enabled | ON | Semi-sync is enabled |
rpl_semi_sync_master_wait_for_slave_count | 1 | One replica ACK is required |
rpl_semi_sync_master_wait_no_slave | ON | Waiting can continue even when too few replicas are connected |
rpl_semi_sync_master_wait_point | AFTER_SYNC | The ACK wait occurs before completing the commit |
rpl_semi_sync_master_timeout | 86400000 | The timeout is 24 hours, in milliseconds |
That last value surprised me. MySQL's usual default timeout is much shorter. MOCO deliberately configures a long timeout to avoid silently falling back to asynchronous replication and weakening its failover-safety assumptions.
It is a deliberate trade-off: protect acknowledged data, even when that means reduced write availability if replicas disappear.
These are MySQL 8.0-style variable names; names and implementation details differ across versions. Always inspect the version running in your cluster.
What we found
The failure state looked like this:
| Instance | Role | Replication state |
|---|---|---|
eks-db-0 | Replica | I/O thread stopped |
eks-db-1 | Primary | Running; replication ACKs unavailable |
eks-db-2 | Replica | I/O thread stopped |
The primary didn't have to crash for applications to suffer. A transaction could reach MySQL and still spend longer than the application's timeout budget waiting for a replica acknowledgement.
A lightweight read-only probe such as SELECT 1 would not necessarily detect that failure. It doesn't exercise the commit path of a real write transaction.
MOCO also distinguishes a cluster with a ready primary from one whose replicas are healthy. Its replica readiness checks include replication status, so Running is not the same as Ready.
One caveat: this was a multi-symptom production incident, not a controlled experiment. Stopped replica threads and semi-sync waits were important evidence; they do not prove that every application timeout had exactly one cause.
What I took away
Semi-synchronous replication is not just a checkbox for high availability. It changes the conditions under which an application write can finish.
For a three-instance MOCO cluster, I'd monitor replica availability and semi-sync wait activity alongside ordinary MySQL uptime. If the last healthy replica disappears, a primary that is still running may no longer be enough.
In Part 4, I'll show the commands I use to investigate that situation: checking MOCO health, inspecting replica threads, reading semi-sync wait counters, and deciding whether a replica can safely rejoin.
