It is sometimes necessary to shutdown or reboot a node to do maintenance tasks, such as to replace hardware, or simply to install a new kernel image. This is also true when using the HA stack. The behaviour of the HA stack during a shutdown can be configured.
Below you will find a description of the different HA policies for a node shutdown. Currently Conditional is the default due to backward compatibility. Some users may find that Migrate behaves more as expected.
Once the Local Resource manager (LRM) gets a shutdown request and this policy is enabled, it will mark itself as unavailable for the current HA manager. This triggers a migration of all HA Services currently located on this node. The LRM will try to delay the shutdown process, until all running services get moved away. But, this expects that the running services can be migrated to another node. In other words, the service must not be locally bound, for example by using hardware passthrough. As non-group member nodes are considered as runnable target if no group member is available, this policy can still be used when making use of HA groups with only some nodes selected. But, marking a group as restricted tells the HA manager that the service cannot run outside of the chosen set of nodes. If all of those nodes are unavailable, the shutdown will hang until you manually intervene. Once the shut down node comes back online again, the previously displaced services will be moved back, if they were not already manually migrated in-between.
The watchdog is still active during the migration process on shutdown. If the node loses quorum it will be fenced and the services will be recovered.
If you start a (previously stopped) service on a node which is currently being maintained, the node needs to be fenced to ensure that the service can be moved and started on another available node.
This mode ensures that all services get stopped, but that they will also be recovered, if the current node is not online soon. It can be useful when doing maintenance on a cluster scale, where live-migrating VMs may not be possible if too many nodes are powered off at a time, but you still want to ensure HA services get recovered and started again as soon as possible.
This mode ensures that all services get stopped and frozen, so that they won’t get recovered until the current node is online again.
The Conditional shutdown policy automatically detects if a shutdown or a reboot is requested, and changes behaviour accordingly.
Shutdown. A shutdown (poweroff) is usually done if it is planned for the node to stay down for some time. The LRM stops all managed services in this case. This means that other nodes will take over those services afterwards.
Recent hardware has large amounts of memory (RAM). So we stop all resources, then restart them to avoid online migration of all that RAM. If you want to use online migration, you need to invoke that manually before you shutdown the node.
Reboot. Node reboots are initiated with the reboot command. This is usually done after installing a new kernel. Please note that this is different from “shutdown”, because the node immediately starts again.
The LRM tells the CRM that it wants to restart, and waits until the CRM puts
all resources into the freeze
state (same mechanism is used for
Package Updates
Section 15.10, “Package Updates”). This prevents those resources
from being moved to other nodes. Instead, the CRM starts the resources after the
reboot on the same node.
Last but not least, you can also manually move resources to other nodes, before you shutdown or restart a node. The advantage is that you have full control, and you can decide if you want to use online migration or not.
Please do not kill services like pve-ha-crm
, pve-ha-lrm
or
watchdog-mux
. They manage and use the watchdog, so this can result in an
immediate node reboot or even reset.