Openstack: How does CapacityWeigher impact which storage pool the Cinder scheduler selects?

How CapacityWeigher works

To better understand how CapacityWeigher works, the following information is key:

1. Per the Cinder Scheduler Weights official documentation:

For thin provisioning, weigh hosts by their virtual free capacity calculated by the total capacity multiplied by the max over subscription ratio and subtracting the provisioned capacity

The formula being used is: total_capacity_gb x max_over_subscription_ratio - provisioned_capacity_gb

2. Provisioned capacity is defined as follows (taken from the provisioning improvements introduced in Queen):

The amount of capacity that would be used in the storage array’s pool being used by Cinder if all the volumes present in there were completely full

CapacityWeigher in action (an example)

Using the above forumla and information, you can take the following event from the Cinder scheduler log and use the values contained within to better understand how weights are determined for each storage pool:

Jul 26 21:10:40 xxx-cinder-api-container-3af405ad cinder-scheduler[34121]: 2022-07-26 21:10:40.353 34121 DEBUG cinder.scheduler.filter_scheduler [req-xxxx] Filtered [host 'xxx@xxx#vol01':free_capacity_gb: 10239.99, total_capacity_gb: 10240.0,allocated_capacity_gb: 0, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300, host 'xxx@xxx#vol02':free_capacity_gb: 10239.99, total_capacity_gb: 10240.0,allocated_capacity_gb: 0, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300, host 'xxx@xxx#vol03':free_capacity_gb: 26258.57, total_capacity_gb: 92160.0,allocated_capacity_gb: 90463, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 90463,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300, host 'xxx@xxx#vol04':free_capacity_gb: 2948.08, total_capacity_gb: 99804.03,allocated_capacity_gb: 99118, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 99118,thin_provisioning_support: True, thick_provisioning_support: False,pools: None,updated at: 2022-07-26 21:10:30.459300] _get_weighted_candidates 

In the above event, we have the following four volumes available to the Cinder scheduler:

vol01:

free_capacity_gb: 10239.99

total_capacity_gb: 10240.0

allocated_capacity_gb: 0

max_over_subscription_ratio: 3.0

provisioned_capacity_gb: 0

vol02:

free_capacity_gb: 10239.99

total_capacity_gb: 10240.0

allocated_capacity_gb: 0

max_over_subscription_ratio: 3.0

provisioned_capacity_gb: 0

vol03:

free_capacity_gb: 26258.57

total_capacity_gb: 92160.0

allocated_capacity_gb: 90463

max_over_subscription_ratio: 3.0

provisioned_capacity_gb: 90463

vol04:

free_capacity_gb: 2948.08

total_capacity_gb: 99804.03

allocated_capacity_gb: 99118

max_over_subscription_ratio: 3.0

reserved_percentage: 0

provisioned_capacity_gb: 99118

 

From the above volumes we can see that:

• vol01 and vol02 are empty.  There is 10TB of available space
• vol03 is considerably larger.  There are 26TB free.  Total capacity is 92TB
• vol04 is nearly out of space.  There are 3 TB free.  Total capacity is 100TB

Even though vol04 is nearly out of space, Cinder scheduler nontheless will select it during cinder volume provisioning:

Jul 26 21:10:40 xxx-cinder-api-container-3af405ad cinder-scheduler[34121]: 2022-07-26 21:10:40.355 34121 DEBUG cinder.scheduler.host_manager [req-xxx] Weighed [WeighedHost [host: xxx@xxx#vol04, weight: 1.0], WeighedHost [host: xxx@xxx#vol03, weight: 0.9158061824185525], WeighedHost [host: xxx@xxx#vol01, weight: 0.0], WeighedHost [host: xxx@xxx#vol02, weight: 0.0]] get_weighed_backends 

To understand this, we can perform the CapacityWeigher calculation (total_capacity_gb x max_over_subscription_ratio - provisioned_capacity_gb):

vol01:

10240 x 3 - 0 = 30720

vol02:

10240 x 3 - 0 = 30720

vol03:

92160 x 3 - 90463 = 186017

vol04:

99804 x 3 - 99118 = 200294

 

In the above calculation, we can see that total_capacity_gb drives the higher weight.  Even though the other volumes have considerably more free capacity, vol04 is still weighted the highest.

What if we increase the size of vol03?

vol03':free_capacity_gb: 36605.8, total_capacity_gb: 102400.0,allocated_capacity_gb: 90563, max_over_subscription_ratio: 3.0,reserved_percentage: 0, provisioned_capacity_gb: 90563

Re-calculating the weights with the new larger vol03:

vol03:

102400 x 3 - 90563 = 216637

vol04:

99804 x 3 - 99113 = 200299

We now see that vol03 has a higher weight.  And as a result, this is the storage pool selected:

Jul 26 21:58:46 usw1infr001-cinder-api-container-3af405ad cinder-scheduler[34121]: 2022-07-26 21:58:46.259 34121 DEBUG cinder.scheduler.host_manager [req-xxx] Weighed [WeighedHost [host: xxx@xxx#vol03, weight: 1.0], WeighedHost [host: xxx@xxx#vol04, weight: 0.9121225600671266], WeighedHost [host: xxx@xxx#vol01, weight: 0.0], WeighedHost [host: xxx@xxx#vol02, weight: 0.0]] get_weighed_backends

 

max_over_subscription_ratio=auto

There is a configurable option that will allow Cinder to adjust how CapacityWeigher handles max_over_subscription_ratio.  For example, if you are unable to have storage pools of all the same total_capacity_gb, than you can configure max_over_subscription_ratio = auto in cinder.conf.  Per the Oversubscription in thin provisioning documentation:

When using auto, Cinder will automatically calculate the max_over_subscription_ratio based on the provisioned capacity and the used space. This allows the creation of a larger number of volumes at the beginning of the pool’s life, and start to restrict the creation as the free space approaches to 0 or the reserved limit.

Takeaways / Conclusions

1.  total_capacity_gb is a signficant factor in the CapacityWeigher calculation
2. When working with thin provisioning volumes, free_capacity_gb is not evaluated
3. Configuring each backing storage pool for a Cinder backend to have similar total_capacity_gb can be an important factor in ensuring that larger storage pools aren't always selected by cinder scheduler.  As the above example illustrates, when working with a nearly full 100TB volume and another empty 10TB volume, the 100TB volume will continue to be weighted higher due to total_capacity_gb
4. If you are unable to size each storage pool the same, consider using max_over_subscription_ratio = auto. This allows Cinder to adjust max_over_subscription_ratio based on provisioned capacity and used space.  This will change the CapacityWeigher calculation and can start favoring smaller storage pools.