Azure CVO panics due to WAFL inconsistencies

• CVO systems deployed with Azure page blob disks are susceptible to I/O throttling from Azure, resulting in “AZURE_SERVER_BUSY” responses
• This throttling can lead to delayed or failed disk operations, causing:
  • Loss of access to critical ONTAP metadata blocks
  • WAFL inconsistencies (especially in root or system volumes)
  • Controller failover issues and node panic events
  • Bad block and checksum verification errors
• Error seen in server logs:

Mon Sep 15 14:27:08 +0000 [jazsacvo100c-01: OscHighPriThreadPool_126: pha.obj.throttle:notice]: PHA Obj throttled 7393 times on the blob 01RootBlob-cvo_systemin the container blobcontainer due to the "AZURE_SERVER_BUSY".

• Contact Azure Support to investigate the issue
• Redeploy the CVO to migrate the instance to new hardware in Azure's data center
  • If this is related to the Azure CVO being deployed with page blobs, the following can be done to mitigate this in the future:
    • Migrate all data to a new CVO deployment that is using Azure managed disks
      • Managed disk-based CVO systems are not subject to “AZURE_SERVER_BUSY” throttling and provide improved performance and reliability
      • This requires deploying a new CVO instance with managed disks and using SnapMirror or other migration tools to move data

• Page blob disks have lower IOPS and throughput limits than managed disks
• Throttling is more likely under high load or with large datasets
• Upsizing the VM instance may not resolve the issue, as page blob limits are at the storage account and disk level
• See: CVO architecture: Page Blob vs Managed Disks