High CIFS access latency occurs under heavy vscan workloads because the AV Connector input queue is becoming full
Applies to
- ONTAP 9
- ONTAP Antivirus Connector (AV Connector)1.0.4
- Trend Micro ServerProtect for Storage 6.0 Build 1119
Issue
- CIFS access becomes very slow under heavy vscan workloads, particularly after running an on-demand vscan.
- The following message appears when a CIFS operation times out because it couldn't be completed within 40 seconds.
Tue Dec 08 11:23:26 CET [Node-01: kernel: Nblade_CifsOperationTimedOut_1:error]: params: {'remoteBladeID': 'xxx-xxxxxxxxxxxx (Node-02)', 'cmdRestartCnt': '0', 'vserverId': '5', 'commandName': 'SMB2_COM_CREATE', 'isQosEnabled': 'QoS_disabled', 'lastSpinNpError': 'SPINNP_ERR_OFFBOX_VSCAN_REQD', 'localIpAddress': '1.1.1.1', 'lastCsmError': 'CSM_OK', 'clientIpAddress': '10.10.10.20', 'suspensionCnt': '6', 'dsId': '0'}
- The AV Connector queue:
scanner_stats_pct_input_queue_avgis keeping a high usage (over 90%) even the on-demand vscan is stopped and vscan workloads are getting decreasing.
Example:
:*> statistics show-periodic -interval 10 -object offbox_vscan_server -counter scanner_stats_pct_input_queue_avg|scan_request_timedout|scan_request_pending|scan_request_dispatched|scanner_stats_is_queue_full -instance svm01:xx.xx.xx.xx:sr-2650-04-01
sr-2650-04: offbox_vscan_server.svm01:10.xx.xx.xx:sr-2650-04-01: 6/3/2021 23:26:12
scan scan scan scanner_stats_is scanner_stats_pct_input
request request request queue queue Complete Number of
dispatched pending timedout full avg Aggregation Constituents
---------- -------- -------- ---------------- ----------------------- ----------- ------------
58 500 3 0 97 Yes 1
53 500 0 0 98 Yes 1
54 500 0 0 98 Yes 1
64 500 0 0 98 Yes 1
55 500 0 0 98 Yes 1
56 500 0 0 98 Yes 1
63 500 0 0 98 Yes 1
55 498 2 0 98 Yes 1
60 500 2 0 98 Yes 1
91 270 261 0 98 Yes 1
56 220 50 0 98 Yes 1
59 220 0 0 98 Yes 1
