Monitoring Patroni: the metrics to watch, failover and replication lag, and five PromQL alerts

Patroni promotes a replica when the leader dies, so a PostgreSQL cluster can look healthy while something is already wrong. Writes fail for a few seconds, a replica falls behind, and nobody notices until the data is needed. This post covers which metrics show those states, how to read failover and replication lag, and five alert rules in PromQL.

The examples use a free Grafana dashboard, PostgreSQL HA Lite (Patroni + postgres_exporter): https://grafana.com/grafana/dashboards/26033/ . Every metric name below comes from the dashboard JSON.

What to scrape

Patroni serves metrics at /metrics on its REST API (Patroni 2.1+, port 8008 in the example). postgres_exporter serves PostgreSQL metrics, usually on port 9187. Use separate jobs for them:

scrape_configs: - job_name: patroni metrics_path: /metrics static_configs: [{targets: ["pg1:8008", "pg2:8008", "pg3:8008"]}] - job_name: postgres static_configs: [{targets: ["pg1:9187", "pg2:9187", "pg3:9187"]}]

Patroni: who is the leader

patroni_primary: 1 on the leader, 0 elsewhere.
patroni_replica: 1 on replicas.
patroni_cluster_unlocked: 1 when the DCS (etcd, Consul, ZooKeeper) has no leader lock.
patroni_postgres_timeline: the current timeline of each node.

A healthy cluster has exactly one leader and the lock is present. Zero leaders means writes fail. Two or more means split brain, which is dangerous, so find the node with the latest data before restarting anything.

Failover and timeline

Every promotion increments the PostgreSQL timeline, so a change in patroni_postgres_timeline reliably marks a switch. The dashboard shows it in two places. "Role per node" plots patroni_primary per node, so a step in the line shows the failover and the new leader. "Timeline changes (24h)" uses changes(patroni_postgres_timeline[24h]), which explains write errors the application saw earlier, even after the service has recovered.

If patroni_cluster_unlocked is 1, the cluster cannot elect a leader even when every node is up.

Replication lag: how much data you could lose

pg_replication_lag_seconds comes from postgres_exporter (replication collector, version 0.15+). Read it on the replicas. If a replica is 60 seconds behind and the leader fails, everything written in those 60 seconds is missing from that replica. Failing over to it loses that data, even though the cluster looks healthy.

The dashboard colors the panel orange from 10 seconds and red from 60 seconds. Tune these to your workload and to how much loss you can accept. If lag grows slowly instead of jumping, look for long transactions or a slow disk on the replica.

Long transactions and connections

pg_stat_activity_max_tx_duration with state!="idle" is the age of the oldest open transaction. It holds back vacuum and pushes replica lag up. The dashboard colors it orange from 5 minutes and red from 30 minutes.
pg_stat_activity_count divided by pg_settings_max_connections is the share of connections in use.

During an incident, check in this order: leader count and lock, then the timeline, then replica lag, then long transactions. That order answers "are writes working", "what changed", "is failover safe", and "why is it slow".

Five alert rules in PromQL

groups: - name: patroni-postgres rules: - alert: PatroniLeaderCountNotOne expr: (count(patroni_primary{job="patroni"} == 1) or vector(0)) != 1 for: 1m labels: severity: critical annotations: summary: "{{ $value }} leaders in the cluster instead of one" - alert: PostgresInstanceDown expr: pg_up{job="postgres"} == 0 for: 2m labels: severity: critical annotations: summary: "postgres_exporter cannot reach {{ $labels.instance }}" - alert: PostgresReplicationLagHigh expr: max by (instance) (pg_replication_lag_seconds{job="postgres"}) > 60 for: 5m labels: severity: warning annotations: summary: "Replica {{ $labels.instance }} is more than 60 s behind" - alert: PatroniTimelineChanged expr: sum(changes(patroni_postgres_timeline{job="patroni"}[15m])) > 0 labels: severity: info annotations: summary: "Leader switched: the timeline changed" - alert: PostgresLongTransaction expr: max by (instance) (pg_stat_activity_max_tx_duration{job="postgres", state!="idle"}) > 1800 for: 5m labels: severity: warning annotations: summary: "A transaction on {{ $labels.instance }} has run for more than 30 minutes"

PatroniLeaderCountNotOne. Fires when the leader count is not exactly one. or vector(0) makes a missing series count as zero, so the alert still fires when no leader is reported. The 1-minute hold ignores the brief flip during a normal failover.
PostgresInstanceDown. pg_up == 0 means postgres_exporter cannot connect. The 2-minute hold avoids noise from a quick restart.
PostgresReplicationLagHigh. The 60-second threshold matches the dashboard's red level. max by (instance) reports the worst replica, not an average that hides it.
PatroniTimelineChanged. Informational. It fires when any timeline changed in the last 15 minutes, so a failover is not missed even if it recovered quickly.
PostgresLongTransaction. The state!="idle" filter skips idle sessions. The 1800-second threshold is the dashboard's 30-minute red level.