Skip to main content
NetApp Knowledge Base

FCP slowness and SRAM dumps during higher workload

Views:
597
Visibility:
Public
Votes:
1
Category:
ontap-9
Specialty:
san
Last Updated:

Applies to

  • ONTAP 9
  • SAN
  • Fibre Channel (FC)

Issue

  • Slowness reported by users that have servers utilizing FCP LUNs
  • Scheduled jobs or planned workload is running in the environment
  • Large deployment of 30+ hosts, 10+ storage nodes
  • QOS volume latency show indicates network is highest contributor of latency
cluster::> qos statistics volume latency show -vserver vserver -volume volume
Workload            ID    Latency    Network    Cluster       Data       Disk    QoS Max    QoS Min      NVRAM      Cloud  FlexCache    SM Sync         VA
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total-              -    24.15ms    22.90ms    15.00us   751.00us   480.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1101.45ms  1101.34ms     6.00us    68.00us    36.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
-total-              -    21.09ms    19.98ms    11.00us   719.00us   375.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
VOLUME            2178   995.61ms   995.50ms     6.00us    49.00us    50.00us        0ms        0ms     1.00us        0ms        0ms        0ms        0ms
-total-              -    26.80ms    25.16ms    12.00us  1206.00us   423.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1107.57ms  1107.45ms     4.00us    69.00us    45.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
-total-              -    22.77ms    21.48ms    13.00us   893.00us   376.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1486.07ms  1485.96ms     6.00us    58.00us    51.00us        0ms        0ms     1.00us        0ms        0ms        0ms        0ms
-total-              -    27.85ms    26.33ms    19.00us  1199.00us   299.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1439.01ms  1438.94ms     3.00us    52.00us     9.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
  • Multiple volumes on multiple storage nodes are impacted, without a pattern
  • SRAM dumps, port resets, threshold reached, and or link break errors are logged intermittently on multiple storage nodes

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_thread_15: scsitarget.fcp.dump:debug]: FCP target SRAM dump generated for adapter 10a, fct_tpd_check_hung_commands: Command termination hung. cmd:0xfffff817e09cdb70 (state=0xa, flags=0x2,ctio_sent=1/1, RecvExAddr=0x11e620, OX_ID=0x4439, RX_ID=0xffff, SID=0x2729e2)

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_thread_15: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 10a.

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO TPD cmd alloc threshold reached handle:7 taskflags:128 Active commands:3891 threshold:3891

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_work_thread_0: scsitarget.ispfct.linkBreak:error]: Link break detected on Fibre Channel target adapter 2a. Firmware status code status1 0x2, status2 0x7, and status4 0x0.

  • Clients have MPIO configured properly with round robin policy and all optimized IO paths are active (two paths per Fabric)
  • Low port discards increment on the switch and SFP power receive/transmit is not at low thresholds

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.