High CPU and latency on FlexGroup volumes
Applies to
- ONTAP 9
- FlexGroups
- NFS
- EDA workloads (can affect other workloads)
- rsync
Issue
- FlexGroups have high Data Processing latency
- CPU utilization is fairly high (above 70 %) on multiple nodes in a cluster
- A
wafl.cp.toolonglong Consistency Point message may be seen during the slowness - FlexGroup space usage may be higher
snc9-naarvrasic01::*> qos statistics volume latency show -node node-202 Workload ID Latency Network Cluster Data Disk QoS Max QoS Min NVRAM Cloud FlexCache SM Sync VA AVSCAN --------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- -total- - 3.48ms 98.00us 119.00us 3.11ms 8.00us 0ms 0ms 142.00us 0ms 0ms 0ms 0ms 0ms vol1-qtree1_3.. 30214 23.89s 241.43ms 0ms 23.50s 0ms 0ms 0ms 144.12ms 0ms 0ms 0ms 0ms 0ms vol2-qtree1_3.. 59604 1041.62ms 82.86ms 0ms 763.67ms 0ms 0ms 0ms 195.10ms 0ms 0ms 0ms 0ms 0ms vol3-qtree1_3.. 12204 646.45ms 44.82ms 0ms 487.15ms 0ms 0ms 0ms 114.47ms 0ms 0ms 0ms 0ms 0ms vol4-qtree1_3.. 15156 499.08ms 95.88ms 0ms 248.38ms 98.85ms 0ms 0ms 55.97ms 0ms 0ms 0ms 0ms 0ms vol5-qtree1_3.. 19951 111.11ms 18.98ms 0ms 77.61ms 0ms 0ms 0ms 14.51ms 0ms 0ms 0ms 0ms 0ms
- Flexgroup show command report increased Urgc and Probabilities values :
::*> system node run -node cluster1 -command flexgroup show -v
FlexGroup 0x8073F3D2 (flexgroup1)
* next snapshot cleanup due in 2091 msec
* next refresh message due in 112 msec (last to member 0x8073F3E0)
* spinnp version negotiated as 13.20, capability 0x1FF7F
* Ref count is 18
* ShouldEnforceQuotas false
* IsAnyMemberInNvfailedState false
* reaction +3.0, workload +1.0, activity level 224, cv 388%, width 1024MB
* Rebalancing configuration:
max_moves=0, max_threshold=0, min_threshold=0
min_file_size=0, max_runtime=0, exclude_snapshots=true
start_time=0, stop_time=0
Idx Member L Used Avail Urgc Target Probabilities D-Ingest Alloc F-Ingest Alloc
--- -------- - --------------- ---------- ---- ------- --------------------- --------- ----- --------- -----
1 1546 L 753661 0% 251147010 0% 6.24% [100% 100% 75% 0%] 0+ 0 0 0+ 0 4
2 1547 R 817846 0% 251143547 0% 6.24% [100% 100% 75% 0%] 0+ 0 0 0+ 0 6
3 1548 L 665116 0% 251154494 0% 6.24% [100% 100% 75% 0%] 0+ 0 0 0+ 0 2
4 1549 R 515989 0% 251265094 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 7
5 1550 L 512874 0% 251265122 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 8
6 1551 R 510543 0% 251265089 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 6
7 1552 L 508497 0% 251265128 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 6
8 1553 R 529048 0% 251265107 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 5
9 1554 L 588224 0% 251269591 25% 6.25% [100% 100% 74% 74%] 0+ 0 0 225+ 108 39 <<<<<<<
10 1555 R 576974 0% 251199550 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 2
11 1556 L 578221 0% 251199649 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 2
12 1557 R 576967 0% 251199532 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 4
13 1558 L 576393 0% 251199504 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 7
14 1559 R 580030 0% 251199472 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 2
15 1560 L 939857 0% 250956106 0% 6.24% [100% 100% 75% 0%] 0+ 0 0 0+ 0 4
16 1561 R 597931 0% 251199456 0% 6.25% [100% 100% 74% 0%] 0+ 0 0 0+ 0 4