Proxmox VE currently uses one of two bootloaders depending on the disk setup selected in the installer.
For EFI Systems installed with ZFS as the root filesystem systemd-boot
is
used. All other deployments use the standard grub
bootloader (this usually
also applies to systems which are installed on top of Debian).
The Proxmox VE installer creates 3 partitions on all disks selected for installation.
The created partitions are:
hdsize
parameter or the remaining space
used for the chosen storage type
Systems using ZFS as root filesystem are booted with a kernel and initrd image
stored on the 512 MB EFI System Partition. For legacy BIOS systems, grub
is
used, for EFI systems systemd-boot
is used. Both are installed and configured
to point to the ESPs.
grub
in BIOS mode (--target i386-pc
) is installed onto the BIOS Boot
Partition of all selected disks on all systems booted with grub
[5].
proxmox-boot-tool
is a utility used to keep the contents of the EFI System
Partitions properly configured and synchronized. It copies certain kernel
versions to all ESPs and configures the respective bootloader to boot from
the vfat
formatted ESPs. In the context of ZFS as root filesystem this means
that you can use all optional features on your root pool instead of the subset
which is also present in the ZFS implementation in grub
or having to create a
separate small boot-pool [6].
In setups with redundancy all disks are partitioned with an ESP, by the installer. This ensures the system boots even if the first boot device fails or if the BIOS can only boot from a particular disk.
The ESPs are not kept mounted during regular operation. This helps to prevent
filesystem corruption to the vfat
formatted ESPs in case of a system crash,
and removes the need to manually adapt /etc/fstab
in case the primary boot
device fails.
proxmox-boot-tool
handles the following tasks:
You can view the currently configured ESPs and their state by running:
# proxmox-boot-tool status
Setting up a new partition for use as synced ESP. To format and initialize a partition as synced ESP, e.g., after replacing a
failed vdev in an rpool, or when converting an existing system that pre-dates
the sync mechanism, proxmox-boot-tool
from pve-kernel-helpers
can be used.
the format
command will format the <partition>
, make sure to pass
in the right device/partition!
For example, to format an empty partition /dev/sda2
as ESP, run the following:
# proxmox-boot-tool format /dev/sda2
To setup an existing, unmounted ESP located on /dev/sda2
for inclusion in
Proxmox VE’s kernel update synchronization mechanism, use the following:
# proxmox-boot-tool init /dev/sda2
Afterwards /etc/kernel/proxmox-boot-uuids
should contain a new line with the
UUID of the newly added partition. The init
command will also automatically
trigger a refresh of all configured ESPs.
Updating the configuration on all ESPs. To copy and configure all bootable kernels and keep all ESPs listed in
/etc/kernel/proxmox-boot-uuids
in sync you just need to run:
# proxmox-boot-tool refresh
(The equivalent to running update-grub
systems with ext4
or xfs
on root).
This is necessary should you make changes to the kernel commandline, or want to sync all kernels and initrds.
Both update-initramfs
and apt
(when necessary) will automatically
trigger a refresh.
Kernel Versions considered by proxmox-boot-tool
. The following kernel versions are configured by default:
Manually keeping a kernel bootable. Should you wish to add a certain kernel and initrd image to the list of
bootable kernels use proxmox-boot-tool kernel add
.
For example run the following to add the kernel with ABI version 5.0.15-1-pve
to the list of kernels to keep installed and synced to all ESPs:
# proxmox-boot-tool kernel add 5.0.15-1-pve
proxmox-boot-tool kernel list
will list all kernel versions currently selected
for booting:
# proxmox-boot-tool kernel list Manually selected kernels: 5.0.15-1-pve Automatically selected kernels: 5.0.12-1-pve 4.15.18-18-pve
Run proxmox-boot-tool kernel remove
to remove a kernel from the list of
manually selected kernels, for example:
# proxmox-boot-tool kernel remove 5.0.15-1-pve
It’s required to run proxmox-boot-tool refresh
to update all EFI System
Partitions (ESPs) after a manual kernel addition or removal from above.
The simplest and most reliable way to determine which bootloader is used, is to watch the boot process of the Proxmox VE node.
You will either see the blue box of grub
or the simple black on white
systemd-boot
.
Determining the bootloader from a running system might not be 100% accurate. The safest way is to run the following command:
# efibootmgr -v
If it returns a message that EFI variables are not supported, grub
is used in
BIOS/Legacy mode.
If the output contains a line that looks similar to the following, grub
is
used in UEFI mode.
Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi)
If the output contains a line similar to the following, systemd-boot
is used.
Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi)
By running:
# proxmox-boot-tool status
you can find out if proxmox-boot-tool
is configured, which is a good
indication of how the system is booted.
grub
has been the de-facto standard for booting Linux systems for many years
and is quite well documented
[7].
Changes to the grub
configuration are done via the defaults file
/etc/default/grub
or config snippets in /etc/default/grub.d
. To regenerate
the configuration file after a change to the configuration run:
[8]
# update-grub
systemd-boot
is a lightweight EFI bootloader. It reads the kernel and initrd
images directly from the EFI Service Partition (ESP) where it is installed.
The main advantage of directly loading the kernel from the ESP is that it does
not need to reimplement the drivers for accessing the storage. In Proxmox VE
proxmox-boot-tool
Section 3.12.2, “Synchronizing the content of the ESP with proxmox-boot-tool
” is used to keep the
configuration on the ESPs synchronized.
systemd-boot
is configured via the file loader/loader.conf
in the root
directory of an EFI System Partition (ESP). See the loader.conf(5)
manpage
for details.
Each bootloader entry is placed in a file of its own in the directory
loader/entries/
An example entry.conf looks like this (/
refers to the root of the ESP):
title Proxmox version 5.0.15-1-pve options root=ZFS=rpool/ROOT/pve-1 boot=zfs linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
You can modify the kernel commandline in the following places, depending on the bootloader used:
Grub. The kernel commandline needs to be placed in the variable
GRUB_CMDLINE_LINUX_DEFAULT
in the file /etc/default/grub
. Running
update-grub
appends its content to all linux
entries in
/boot/grub/grub.cfg
.
Systemd-boot. The kernel commandline needs to be placed as one line in /etc/kernel/cmdline
.
To apply your changes, run proxmox-boot-tool refresh
, which sets it as the
option
line for all config files in loader/entries/proxmox-*.conf
.
[5] These are all installs with root on ext4
or xfs
and installs
with root on ZFS on non-EFI systems
[6] Booting ZFS on root with grub https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS
[8] Systems using proxmox-boot-tool
will call proxmox-boot-tool
refresh
upon update-grub
.