This section describes a way to deploy an external voter in a Proxmox VE cluster. When configured, the cluster can sustain more node failures without violating safety properties of the cluster communication.
For this to work, there are two services involved:
As a result, you can achieve higher availability, even in smaller setups (for example 2+1 nodes).
The Corosync Quorum Device (QDevice) is a daemon which runs on each cluster node. It provides a configured number of votes to the cluster’s quorum subsystem, based on an externally running third-party arbitrator’s decision. Its primary use is to allow a cluster to sustain more node failures than standard quorum rules allow. This can be done safely as the external device can see all nodes and thus choose only one set of nodes to give its vote. This will only be done if said set of nodes can have quorum (again) after receiving the third-party vote.
Currently, only QDevice Net is supported as a third-party arbitrator. This is a daemon which provides a vote to a cluster partition, if it can reach the partition members over the network. It will only give votes to one partition of a cluster at any time. It’s designed to support multiple clusters and is almost configuration and state free. New clusters are handled dynamically and no configuration file is needed on the host running a QDevice.
The only requirements for the external host are that it needs network access to the cluster and to have a corosync-qnetd package available. We provide a package for Debian based hosts, and other Linux distributions should also have a package available through their respective package manager.
In contrast to corosync itself, a QDevice connects to the cluster over TCP/IP. The daemon may even run outside of the cluster’s LAN and can have longer latencies than 2 ms.
We support QDevices for clusters with an even number of nodes and recommend it for 2 node clusters, if they should provide higher availability. For clusters with an odd node count, we currently discourage the use of QDevices. The reason for this is the difference in the votes which the QDevice provides for each cluster type. Even numbered clusters get a single additional vote, which only increases availability, because if the QDevice itself fails, you are in the same position as with no QDevice at all.
On the other hand, with an odd numbered cluster size, the QDevice provides (N-1) votes — where N corresponds to the cluster node count. This alternative behavior makes sense; if it had only one additional vote, the cluster could get into a split-brain situation. This algorithm allows for all nodes but one (and naturally the QDevice itself) to fail. However, there are two drawbacks to this:
If you understand the drawbacks and implications, you can decide yourself if you want to use this technology in an odd numbered cluster setup.
We recommend running any daemon which provides votes to corosync-qdevice as an unprivileged user. Proxmox VE and Debian provide a package which is already configured to do so. The traffic between the daemon and the cluster must be encrypted to ensure a safe and secure integration of the QDevice in Proxmox VE.
First, install the corosync-qnetd package on your external server
external# apt install corosync-qnetd
and the corosync-qdevice package on all cluster nodes
pve# apt install corosync-qdevice
After doing this, ensure that all the nodes in the cluster are online.
You can now set up your QDevice by running the following command on one of the Proxmox VE nodes:
pve# pvecm qdevice setup <QDEVICE-IP>
The SSH key from the cluster will be automatically copied to the QDevice.
Make sure that the SSH configuration on your external server allows root login via password, if you are asked for a password during this step.
After you enter the password and all the steps have successfully completed, you will see "Done". You can verify that the QDevice has been set up with:
pve# pvecm status ... Votequorum information ~~~~~~~~~~~~~~~~~~~~~ Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags: Quorate Qdevice Membership information ~~~~~~~~~~~~~~~~~~~~~~ Nodeid Votes Qdevice Name 0x00000001 1 A,V,NMW 192.168.22.180 (local) 0x00000002 1 A,V,NMW 192.168.22.181 0x00000000 1 Qdevice
In case of a tie, where two same-sized cluster partitions cannot see each other but can see the QDevice, the QDevice chooses one of those partitions randomly and provides a vote to it.
For clusters with an even node count, there are no negative implications when using a QDevice. If it fails to work, it is the same as not having a QDevice at all.
If you want to add a new node or remove an existing one from a cluster with a QDevice setup, you need to remove the QDevice first. After that, you can add or remove nodes normally. Once you have a cluster with an even node count again, you can set up the QDevice again as described previously.