What is Adaptive QoS and how does it work?

Adaptive QoS uses QoS Throughput Floors and Ceilings (Minimum and Maximum throttle limits) to set individual Volume limits.
- Floors are used to prevent bully workloads from robbing workloads of their share of resources
  - This keeps workloads from going below a minimum amount of IOPS and/or MB/s
  - Think of AQoS as the inverse of a ceiling or traditional QoS policy: it keeps other workloads from taking resources from the workload the policy is applied to
- Ceilings are used to limit busier workloads to prevent robbing resources from other workloads
Adaptive QoS is dynamic based upon volume size, meaning a 10GB volume would have a different Floor and Ceiling than a 10 TB volume.
- This means the Ceiling is the greater of 1) Expected IOPS, 2) Peak IOPS, or 3) Absolute Minimum IOPS
  - Note: Expected or peak could be determined by if used or allocated space is set, so if calculating, please consider that
- The Floor is always Expected IOPS unless below the Absolute Minimum IOPS
As with regular QoS, AQoS is a cluster wide process as I/O may hit any LIF on any node in the Cluster

Adaptive QoS	Dynamic QoS Ceilings and Floors which grow or shrink based upon volume size used or allocated
Throughput Floor (Minimum)	A guaranteed throughput measurement (IOPS and/or MB/s) that gives non-floored workloads less priority over floored workloads
Throughput Ceiling (Maximum)	A hard limit of how many IOPs a Volume is allocated (regular QoS)
Expected IOPS	The Throughput Floor value in IOPS per terabyte (unless specified different)
Absolute minimum IOPS	An IOP Throughput Floor, used when Expected IOPs becomes too low and overrides Expected IOPs or Peak Example: A 10 GB Volume with a default Adaptive QoS "value" policy group will have a Floor of 75 IOPs, not 1.28 IOPs Expected.
Peak IOPS	The value in IOPS per terabyte (unless specified different) is the Ceiling when greater than Expected and Absolute Minimum IOPS
Allocated Space	AQoS limits can be set on this Commands which show the calculated value: `volume show -fields total` `df -g` not including the `.snapshot` line
Used Space	This is the logical space including snapshot spill and excluding storage-efficiency space savings Command which show the calculated value: `volume show -fields logical used`
Headroom	The calculated value of optimal performance on CPU (and disk) See more on the KB Is my controller overloaded?

Note: The calculated value can be seen with the qos workload show -instance command

Cluster::> qos workload show -instance

          Workload Name: aqos1-wid32444
...
     Maximum Throughput: 1425IOPS

Custom policies may be made available with the qos adaptive-policy-group create command
By default, three buckets are created:

policy-group	expected-iops	peak-iops	absolute-min-iops	peak-iops-allocation
extreme	6144IOPS/TB	12288IOPS/TB	1000IOPS	used-space
performance	2048IOPS/TB	4096IOPS/TB	500IOPS	used-space
value	128IOPS/TB	512IOPS/TB	75IOPS	used-space

IOPS from volumes without a floor will be put into a queue Best Effort
IOPS from volumes with a floor will be put into a deadline queue
IOPS from volumes with floors get priority over volumes without floors in the dblade as long as workload on those volumes is under the floor value.
IOPS from volumes exceeding the floor are treated the same as a volume without a floor.
- IOPS queue in a delay center called QoS Minimums
- IOPS in this queue are treated along with those from volumes without a floor in the Best Effort queue
If CPU is above the headroom value:
1. Below the CPU headroom optimal point, Best Effort IOPS are yielded to deadline IOPS
2. Above the CPU headroom optimal point on ONTAP 9.6 and below, the deadline IOPS may be lower but will still get priority over Best Effort IOPS
3. Above the CPU headroom optimal point on ONTAP 9.7 and above, deadline IOPS will get the same values as below the optimal point and Best Effort IOPS are throttled heavier
IOPS from volumes hitting the Throughput Ceiling will be hard throttled at that value.

Note: ONTAP 9.7 changes are due to Throughput Floors v2.

Supported QoS features per ONTAP version
- The link takes you to General Support but there are a few sections below that which have more tables of supported features for QoS and AQoS.
Documentation on Adaptive QoS.
Documentation on Throughput Floors
What is Quality of Service (QoS) in ONTAP?

Volume Size	Data Stored	QoS Min IOPS (SSD Aggregate only)	QoS Max IOPS
1 GB	0GB	75 (Absolute Minimum)	75 (Absolute Minimum)
1 TB	0 TB	128 (Expected)	128 (Expected)
1 TB	.1 TB	128 (Expected)	128 (Expected)
1 TB	.2 TB	128 (Expected)	128 (Expected)
1 TB	.3 TB	128 (Expected)	154 (Peak)
1 TB	.5 TB	128 (Expected)	256 (Peak)
1 TB	1 TB	128 (Expected)	512 (Peak)
2 TB	2 TB	256 (Expected)	1024 (Peak)