Skip to main content
NetApp Knowledge Base

CHW-3859: Very poor performance and high latency with shelf module storage port down

  1. Last updated
  2. Save as PDF
Views:
2
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
hw
Last Updated:

Issue

  • Significant performance degradation impacting specific data:
    • For ALL servers/clients accessing that data.
    • With high latency spikes for a specific period of time. 
  • That data is linked to a:
    • Specific SVM(s) and volume(s)
    • That SVM(s) and volume(s) are in the same aggregate(s)
    • That aggregate(s) is using drives in a single common shelf.
  • The drives in that shelf report ioLatency exceed messages evenly distributed across the time the issue is noticed. Example:

[node_name: disk_latency_monitor: shm.ssd.threshold.ioLatency:notice]: SSD e1a.2.3.0 has exceeded the expected block latency in the current timeframe with an average latency of 8450 us and an average utilization of 50 percent. The next highest SSD latency: 8450 us. Disk e1a.2.3.0 Shelf 3 Bay 0 [NETAPP X4...
  • Those drives are being power cycle. Example:

[node_name: scsi_cmdblk_strthr_admin: disk.timeout.flush.start:notice]: Aggressive timeout flush started on disk e1a.2.3.0 S/N S... Details: Power Cycled: true Reason: Disk Spasm.
[node_name: disk_admin: disk.timeout.flush.end:notice]: Aggressive timeout flush ended on disk e1a.2.3.0 S/N S...
  • Multiple wafl.cp.toolong events observed on the node for that aggregate. Example: 

[node_name: wafl_exempt07: wafl.cp.toolong:error]: Aggregate aggr_name experienced a long CP. 
  • The issue stops with a shelf port down event on one of the storage ports for a module in that shelf. Example: 

[node_name: dsa_worker5: ses.status.etherConn.warn:error]: NS224NSM100 (S/N SH...) shelf 2 on channel 0x Ethernet connector warning for port e0b: a cable has been removed. This element is on the rear of the shelf at the bottom, on module B.
  • Latency reported in Grafana Disk View for the node and aggregate during that period of time

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.