8.6. Ceph Pools

screenshot/gui-ceph-pools.png

A pool is a logical group for storing objects. It holds a collection of objects, known as Placement Groups (PG, pg_num).

You can create and edit pools from the command line or the web-interface of any Proxmox VE host under Ceph → Pools.

When no options are given, we set a default of 128 PGs, a size of 3 replicas and a min_size of 2 replicas, to ensure no data loss occurs if any OSD fails.

It is advised that you either enable the PG-Autoscaler or calculate the PG number based on your setup. You can find the formula and the PG calculator [17] online. From Ceph Nautilus onward, you can change the number of PGs [18] after the setup.

The PG autoscaler [19] can automatically scale the PG count for a pool in the background. Setting the Target Size or Target Ratio advanced parameters helps the PG-Autoscaler to make better decisions.

Example for creating a pool over the CLI. 

pveceph pool create <name> --add_storages

Tip

If you would also like to automatically define a storage for your pool, keep the ‘Add as Storage’ checkbox checked in the web-interface, or use the command line option --add_storages at pool creation.

screenshot/gui-ceph-pool-create.png

The following options are available on pool creation, and partially also when editing a pool.

Name
The name of the pool. This must be unique and can’t be changed afterwards.
Size
The number of replicas per object. Ceph always tries to have this many copies of an object. Default: 3.
PG Autoscale Mode
The automatic PG scaling mode [19] of the pool. If set to warn, it produces a warning message when a pool has a non-optimal PG count. Default: warn.
Add as Storage
Configure a VM or container storage using the new pool. Default: true (only visible on creation).

Advanced Options

Min. Size
The minimum number of replicas per object. Ceph will reject I/O on the pool if a PG has less than this many replicas. Default: 2.
Crush Rule
The rule to use for mapping object placement in the cluster. These rules define how data is placed within the cluster. See Ceph CRUSH & device classes Section 8.7, “Ceph CRUSH & device classes” for information on device-based rules.
# of PGs
The number of placement groups [18] that the pool should have at the beginning. Default: 128.
Target Ratio
The ratio of data that is expected in the pool. The PG autoscaler uses the ratio relative to other ratio sets. It takes precedence over the target size if both are set.
Target Size
The estimated amount of data expected in the pool. The PG autoscaler uses this size to estimate the optimal PG count.
Min. # of PGs
The minimum number of placement groups. This setting is used to fine-tune the lower bound of the PG count for that pool. The PG autoscaler will not merge PGs below this threshold.

Further information on Ceph pool handling can be found in the Ceph pool operation [20] manual.

To destroy a pool via the GUI, select a node in the tree view and go to the Ceph → Pools panel. Select the pool to destroy and click the Destroy button. To confirm the destruction of the pool, you need to enter the pool name.

Run the following command to destroy a pool. Specify the -remove_storages to also remove the associated storage.

pveceph pool destroy <name>

The PG autoscaler allows the cluster to consider the amount of (expected) data stored in each pool and to choose the appropriate pg_num values automatically. It is available since Ceph Nautilus.

You may need to activate the PG autoscaler module before adjustments can take effect.

ceph mgr module enable pg_autoscaler

The autoscaler is configured on a per pool basis and has the following modes:

warn

A health warning is issued if the suggested pg_num value differs too much from the current value.

on

The pg_num is adjusted automatically with no need for any manual interaction.

off

No automatic pg_num adjustments are made, and no warning will be issued if the PG count is not optimal.

The scaling factor can be adjusted to facilitate future data storage with the target_size, target_size_ratio and the pg_num_min options.

You can find a more in-depth introduction to the PG autoscaler on Ceph’s Blog - New in Nautilus: PG merging and autotuning.