High availability in Proxmox VE: HA resources, fencing and affinity rules

High availability (HA) means the cluster watches selected guests and, if the node running one of them fails, starts the guest again on a healthy node. Proxmox VE’s ha-manager acts like an automated administrator. It does not make a failure invisible: the guest is restarted, so it reboots. Typical detection and recovery takes about two minutes.

Requirements

• At least three cluster nodes, so that the remaining nodes still have quorum after one fails. Two nodes plus a QDevice also give three votes.
• Shared storage for HA guests (for example Ceph, NFS or a shared SAN), or storage replication, so another node can access their disks.
• Redundant hardware and reliable networking. HA reacts to failures; it cannot fix a flaky network, and an unstable cluster network can cause unexpected node resets.
• Guests should only use resources available on all nodes they can run on: same bridges, storage and passthrough devices.

Fencing: why nodes reboot themselves

Before a guest can be started elsewhere, the cluster must be sure the failed node is really not running it any more. If both copies wrote to the same shared disk, the data would be destroyed. Making sure a node is offline is called fencing.

Proxmox VE uses self-fencing with watchdog timers. While a node is healthy and in the quorate part of the cluster, its HA services keep resetting a watchdog. If the node loses quorum while it runs HA services, it stops resetting the watchdog, and after the timeout (60 seconds) the watchdog resets the whole server. Only after that point do the other nodes recover the guests. A hardware watchdog is used if one is configured; otherwise the Linux kernel’s softdog is used.

This is why a node that loses its cluster network may suddenly reboot when HA is in use. It is intentional: it protects your data.

How it works

• pve-ha-lrm (local resource manager) runs on every node and starts or stops the HA services assigned to it.
• pve-ha-crm (cluster resource manager) runs on every node, but only one is the active master at a time. It makes cluster-wide decisions, such as where to recover a service after fencing.
• Locks in the cluster file system ensure only one master exists and that a failed node’s services can safely be taken over.

HA resources

Each managed guest is an HA resource with an ID such as vm:100 or ct:101. You add and control them in the web UI under Datacenter → HA, or on the command line:

ha-manager add vm:100
ha-manager set vm:100 --state stopped
ha-manager set vm:100 --state started
ha-manager status
ha-manager migrate vm:100 pve2     # live migration
ha-manager relocate vm:100 pve2    # stop, move, start
ha-manager remove vm:100

Once a guest is under HA, normal commands such as qm start 100 or qm stop 100 only change its requested state, and the HA stack carries it out a few seconds later. Options --max_restart and --max_relocate (default 1 each) control how often HA retries starting a failing service on the same node or on another node.

Placement: groups and affinity rules

Older versions used HA groups to limit where a resource may run. Since Proxmox VE 9.0, groups are deprecated and migrated to HA rules, stored in /etc/pve/ha/rules.cfg:

• Node affinity rules prefer (or, with --strict 1, require) certain nodes, optionally with priorities, or avoid nodes with negative affinity.
• Resource affinity rules keep resources together on one node (positive) or on separate nodes (negative), for example two database replicas that should never share a host.

ha-manager rules add node-affinity prefer-pve1 --resources vm:100 --nodes pve1
ha-manager rules set node-affinity prefer-pve1 --strict 1
ha-manager rules add resource-affinity keep-apart --affinity negative --resources vm:201,vm:202

On older releases, use the HA groups described in that version’s documentation instead.

Maintenance

To work on a node without HA fighting you, put it into maintenance mode: ha-manager crm-command node-maintenance enable pve2. HA migrates its services to other nodes and moves them back after node-maintenance disable pve2. Maintenance mode survives reboots until you disable it. Always check ha-manager status and pvecm status before rebooting nodes in an HA cluster, and reboot one node at a time.

Official documentation

• ha-manager(1): High Availability
• pvecm(1): Cluster Manager (QDevice and quorum)