Openstack: How does CapacityWeigher impact which storage pool the Cinder scheduler selects?
Applies to
- Openstack
Answer
When there are multiple thin-provisioned storage pools available, the Cinder driver leverages CapacityWeigher to determine which storage pool to provision to.
How CapacityWeigher works
To better understand how CapacityWeigher works, the following information is key:
- Per the Cinder Scheduler Weights official documentation:
For thin provisioning, weigh hosts by their virtual free capacity calculated by the total capacity multiplied by the max over subscription ratio and subtracting the provisioned capacity
The formula being used is: total_capacity_gb x max_over_subscription_ratio - provisioned_capacity_gb
- Provisioned capacity is defined as follows (taken from the provisioning improvements introduced in Queen):
The amount of capacity that would be used in the storage array’s pool being used by Cinder if all the volumes present in there were completely full
CapacityWeigher in action (an example)
Using the above forumla and information, you can take the following event from the Cinder scheduler log and use the values contained within to better understand how weights are determined for each storage pool:
Jul 26 21:10:40 xxx-cinder-api-container-3af405ad cinder-scheduler[34121]: 2022-07-26 21:10:40.353 34121 DEBUG cinder.scheduler.filter_scheduler [req-xxxx] Filtered [host 'xxx@xxx#vol01':free_capacity_gb: 10239.99, total_capacity_gb: 10240.0,allocated_capacity_gb: 0, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300, host 'xxx@xxx#vol02':free_capacity_gb: 10239.99, total_capacity_gb: 10240.0,allocated_capacity_gb: 0, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300, host 'xxx@xxx#vol03':free_capacity_gb: 26258.57, total_capacity_gb: 92160.0,allocated_capacity_gb: 90463, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 90463,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300, host 'xxx@xxx#vol04':free_capacity_gb: 2948.08, total_capacity_gb: 99804.03,allocated_capacity_gb: 99118, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 99118,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300] _get_weighted_candidates
In the above event, we have the following four volumes available to the Cinder scheduler:
vol01:
free_capacity_gb: 10239.99
total_capacity_gb: 10240.0
allocated_capacity_gb: 0
max_over_subscription_ratio: 3.0
provisioned_capacity_gb: 0
vol02:
free_capacity_gb: 10239.99
total_capacity_gb: 10240.0
allocated_capacity_gb: 0
max_over_subscription_ratio: 3.0
provisioned_capacity_gb: 0
vol03:
free_capacity_gb: 26258.57
total_capacity_gb: 92160.0
allocated_capacity_gb: 90463
max_over_subscription_ratio: 3.0
provisioned_capacity_gb: 90463
vol04:
free_capacity_gb: 2948.08
total_capacity_gb: 99804.03
allocated_capacity_gb: 99118
max_over_subscription_ratio: 3.0
reserved_percentage: 0
provisioned_capacity_gb: 99118
From the above volumes we can see that:
- vol01 and vol02 are empty. There is 10TB of available space
- vol03 is considerably larger. There are 26TB free. Total capacity is 92TB
- vol04 is nearly out of space. There are 3 TB free. Total capacity is 100TB
Even though vol04 is nearly out of space, Cinder scheduler nontheless will select it during cinder volume provisioning:
Jul 26 21:10:40 xxx-cinder-api-container-3af405ad cinder-scheduler[34121]: 2022-07-26 21:10:40.355 34121 DEBUG cinder.scheduler.host_manager [req-xxx] Weighed [WeighedHost [host: xxx@xxx#vol04, weight: 1.0], WeighedHost [host: xxx@xxx#vol03, weight: 0.9158061824185525], WeighedHost [host: xxx@xxx#vol01, weight: 0.0], WeighedHost [host: xxx@xxx#vol02, weight: 0.0]] get_weighed_backends
To understand this, we can perform the CapacityWeigher calculation (total_capacity_gb x max_over_subscription_ratio - provisioned_capacity_gb):
vol01:
10240 x 3 - 0 = 30720
vol02:
10240 x 3 - 0 = 30720
vol03:
92160 x 3 - 90463 = 186017
vol04:
99804 x 3 - 99118 = 200294
In the above calculation, we can see that total_capacity_gb
drives the higher weight. Even though the other volumes have considerably more free capacity, vol04 is still weighted the highest.
What if we increase the size of vol03?
vol03':free_capacity_gb: 36605.8, total_capacity_gb: 102400.0,allocated_capacity_gb: 90563, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 90563
Re-calculating the weights with the new larger vol03:
vol03:
102400 x 3 - 90563 = 216637
vol04:
99804 x 3 - 99113 = 200299
We now see that vol03 has a higher weight. And as a result, this is the storage pool selected:
Jul 26 21:58:46 usw1infr001-cinder-api-container-3af405ad cinder-scheduler[34121]: 2022-07-26 21:58:46.259 34121 DEBUG cinder.scheduler.host_manager [req-xxx] Weighed [WeighedHost [host: xxx@xxx#vol03, weight: 1.0], WeighedHost [host: xxx@xxx#vol04, weight: 0.9121225600671266], WeighedHost [host: xxx@xxx#vol01, weight: 0.0], WeighedHost [host: xxx@xxx#vol02, weight: 0.0]] get_weighed_backends
max_over_subscription_ratio=auto
There is a configurable option that will allow Cinder to adjust how CapacityWeigher handles max_over_subscription_ratio
. For example, if you are unable to have storage pools of all the same total_capacity_gb
, than you can configure max_over_subscription_ratio = auto
in cinder.conf. Per the Oversubscription in thin provisioning documentation:
When using auto, Cinder will automatically calculate the
max_over_subscription_ratio
based on the provisioned capacity and the used space. This allows the creation of a larger number of volumes at the beginning of the pool’s life, and start to restrict the creation as the free space approaches to 0 or the reserved limit.
Takeaways / Conclusions
-
total_capacity_gb
is a signficant factor in the CapacityWeigher calculation - When working with thin provisioning volumes,
free_capacity_gb
is not evaluated - Configuring each backing storage pool for a Cinder backend to have similar
total_capacity_gb
can be an important factor in ensuring that larger storage pools aren't always selected by cinder scheduler. As the above example illustrates, when working with a nearly full 100TB volume and another empty 10TB volume, the 100TB volume will continue to be weighted higher due tototal_capacity_gb
- If you are unable to size each storage pool the same, consider using
max_over_subscription_ratio = auto
. This allows Cinder to adjustmax_over_subscription_ratio
based on provisioned capacity and used space. This will change the CapacityWeigher calculation and can start favoring smaller storage pools.
Additional Information