Patroni cluster architecture

A typical Patroni cluster has these components:

  • Patroni (Python daemon): runs on each PostgreSQL node. Manages the PostgreSQL lifecycle, competes for leadership in the DCS and coordinates failover.
  • DCS (Distributed Configuration Store): etcd, Consul or ZooKeeper. Acts as the sole arbiter of who is the leader. Must have at least 3 nodes for fault tolerance.
  • HAProxy: balancer that detects the current leader via Patroni's REST endpoint (/primary and /replica) and distributes connections.
  • PgBouncer: connection pooler that sits between the application and HAProxy, reusing PostgreSQL connections and reducing max_connections load.
  • Watchdog: OS device (/dev/watchdog) that restarts the server if Patroni stops feeding it, preventing split-brain.

Symptoms indicating cluster problems

  • patronictl list shows a node with role Replica but state stopped or unknown
  • patronictl list shows no node as Leader
  • The application receives connection errors during an unexpected failover
  • Patroni logs show "DCS is not accessible" or "failover is not allowed during pause"
  • The replication lag between the leader and replicas exceeds the safety threshold
  • HAProxy shows all backends in DOWN state

Business risks

  • A misconfigured failover can result in split-brain and data loss or corruption
  • If the DCS has no quorum, the cluster pauses and accepts no failover — availability depends on the DCS
  • A very delayed replica may be promoted with incomplete data if the leader fails
  • If PgBouncer is not configured with auto-reconnect, sessions don't recover on their own

Configuration and operations checklist

  • 1. The DCS has at least 3 nodes for quorum With 2 nodes or 1 single etcd, DCS failure completely paralyzes the cluster. The minimum recommendation is 3 etcd nodes.
  • 2. TTL and loop_wait are correctly configured ttl must be greater than loop_wait * 2. A TTL that is too low generates spurious failovers; too high delays failure detection.
  • 3. Watchdog enabled to prevent split-brain Without watchdog, a node that loses connection to the DCS could continue accepting writes while a replica is being promoted.
  • 4. HAProxy configured with healthcheck to the Patroni endpoint
  • 5. maximum_lag_on_failover configured Prevents a very delayed replica from being automatically promoted. If the lag in bytes exceeds this value, the replica is not eligible.
  • 6. The application handles reconnections after a failover In a failover, active connections are cut. The application must retry the connection. PgBouncer with an appropriate reconnect_timeout helps absorb the impact.
  • 7. Periodically verify replication lag between nodes
  • 8. Perform a periodic planned switchover A cluster that has never been switched may fail at the critical moment for unforeseen reasons. Scheduled switchovers verify everything works and train the team.
TERMINAL 
# Cluster status and who the current leader is
patronictl -c /etc/patroni.yml list

When to escalate to a specialist DBA

  • The cluster entered pause state due to DCS loss and the primary remains active with no possible failover
  • There is evidence of split-brain: two nodes that simultaneously believe they are the leader
  • Automatic failover did not occur after the leader died and there is prolonged downtime
  • There is data loss after an unplanned failover (timeline divergence)
  • The cluster has a "Replica" node that cannot rejoin the cluster after a pause

Frequently asked questions

What happens to the cluster if the DCS (etcd) goes down?

Patroni enters pause mode: it makes no changes to the cluster, allows no automatic failover and cannot elect a new leader until the DCS recovers. The current leader may continue serving write traffic but cannot confirm its leadership. This behavior is intentional to prevent split-brain. That is why etcd must have at least 3 nodes (tolerance to 1 failure) or 5 nodes (tolerance to 2 failures) to guarantee quorum.

What is split-brain in Patroni and how is it prevented?

Split-brain occurs when two nodes simultaneously believe they are the cluster leader, both accept writes and data diverges — a situation that can be irrecoverable. Patroni prevents this using the DCS as the sole arbiter of who holds the leader lock. Additionally, the OS watchdog restarts or shuts down the node that loses connection to the DCS, ensuring there are never two active primaries at the same time.

How long does an automatic failover take?

Typically between 15 and 45 seconds from when Patroni detects the leader's failure to when the most advanced replica is promoted and HAProxy redirects traffic. The time depends on the ttl value configured in the DCS (time for the leadership lock to expire), the DCS response speed, and the time the replica takes to complete its promotion. Typical production values: ttl=30s, loop_wait=10s.

Can Patroni be used with cloud PostgreSQL instances?

Yes. Patroni works well on EC2, GCE, Azure VMs and on-premises environments. For Kubernetes there is the Patroni operator (Zalando Postgres Operator) that manages Patroni clusters as native Kubernetes resources. The DCS can be external etcd, a managed etcd (like EKS's) or Consul. The main consideration is that the DCS has low latency with the PostgreSQL nodes.

Related resources