Slow performance with vscan enabled due to overloaded or disconnected AV Scanner/s
Applies to
- ONTAP 9
- Virus Scan (vScan)
- Trend Micro
- Symantec
- Trellix
Issue
- The latency is listed high which is not accounted for in any storage subsystem
cluster::> qos statistics workload latency show Workload ID Latency Network Cluster Data Disk QoS Max QoS Min NVRAM Cloud FlexCache SM Sync VA --------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- vol1 4040 64.16ms 1.72ms 0ms 57.00us 14.00us 1ms 0ms 1.00us 0ms 0ms 0ms 0ms vol2 6053 49.26ms 1.21ms 35.00us 870.00us 106.00us 0ms 0ms 2.00us 0ms 0ms 0ms 0ms
- Slow performance due to High latency in CIFS introduced by Vscan
Example:
::*> statistics show -object offbox_vscan_server -instance * -counter scan_latency|scan_latency_base -sample-id vscan_test1 -sort-order descending -sort-key scan_latency -max 10
Object: offbox_vscan_server
Instance: ...
Start-time: 3/10/2020 04:56:22
End-time: 3/10/2020 05:58:29
Elapsed-time: 3722s
[...]
Counter Value
-------------------------------- --------------------------------
scan_latency 456294us
scan_latency_base 372867
Note: scan_latency_base counts requests made as a base for the average scan_latency
- This message occurs when a CIFS operation times out because it could not be processed completely within 40 seconds
Tue Dec 08 11:23:26 CET [Node-01: kernel: Nblade_CifsOperationTimedOut_1:error]: params: {'remoteBladeID': 'xxx-xxxxxxxxxxxx (Node-02)', 'cmdRestartCnt': '0', 'vserverId': '5', 'commandName': 'SMB2_COM_CREATE', 'isQosEnabled': 'QoS_disabled', 'lastSpinNpError': 'SPINNP_ERR_OFFBOX_VSCAN_REQD', 'localIpAddress': '10.10.10.10', 'lastCsmError': 'CSM_OK', 'clientIpAddress': '10.10.10.20', 'suspensionCnt': '6', 'dsId': '0'}
- Networking issues contributing to significant latency to vscan server ruled out