8.5. Ceph OSDs

screenshot/gui-ceph-osd-status.png

Ceph Object Storage Daemons store objects for Ceph over the network. It is recommended to use one OSD per physical disk.

You can create an OSD either via the Proxmox VE web-interface or via the CLI using pveceph. For example:

pveceph osd create /dev/sd[X]

If the disk was in use before (for example, for ZFS or as an OSD) you first need to zap all traces of that usage. To remove the partition table, boot sector and any other OSD leftover, you can use the following command:

ceph-volume lvm zap /dev/sd[X] --destroy

Ceph Bluestore. Starting with the Ceph Kraken release, a new Ceph OSD storage type was introduced called Bluestore [16]. This is the default when creating OSDs since Ceph Luminous.

pveceph osd create /dev/sd[X]

Block.db and block.wal. If you want to use a separate DB/WAL device for your OSDs, you can specify it through the -db_dev and -wal_dev options. The WAL is placed with the DB, if not specified separately.

pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z]

You can directly choose the size of those with the -db_size and -wal_size parameters respectively. If they are not given, the following values (in order) will be used:

  • bluestore_block_{db,wal}_size from Ceph configuration…

    • … database, section osd
    • … database, section global
    • … file, section osd
    • … file, section global
  • 10% (DB)/1% (WAL) of OSD size

Note

The DB stores BlueStore’s internal metadata, and the WAL is BlueStore’s internal journal or write-ahead log. It is recommended to use a fast SSD or NVRAM for better performance.

Ceph Filestore. Before Ceph Luminous, Filestore was used as the default storage type for Ceph OSDs. Starting with Ceph Nautilus, Proxmox VE does not support creating such OSDs with pveceph anymore. If you still want to create filestore OSDs, use ceph-volume directly.

ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]

To remove an OSD via the GUI, first select a Proxmox VE node in the tree view and go to the Ceph → OSD panel. Then select the OSD to destroy and click the OUT button. Once the OSD status has changed from in to out, click the STOP button. Finally, after the status has changed from up to down, select Destroy from the More drop-down menu.

To remove an OSD via the CLI run the following commands.

ceph osd out <ID>
systemctl stop ceph-osd@<ID>.service

The following command destroys the OSD. Specify the -cleanup option to additionally destroy the partition table.

pveceph osd destroy <ID>