FCP slowness and SRAM dumps during higher workload

Applies to

• ONTAP 9
• SAN
• Fibre Channel (FC)

Issue

• Slowness reported by users that have servers utilizing FCP LUNs
• Scheduled jobs or planned workload is running in the environment
• Large deployment of 30+ hosts, 10+ storage nodes
• QOS volume latency show indicates network is highest contributor of latency

cluster::> qos statistics volume latency show -vserver vserver -volume volume
Workload            ID    Latency    Network    Cluster       Data       Disk    QoS Max    QoS Min      NVRAM      Cloud  FlexCache    SM Sync         VA
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total-              -    24.15ms    22.90ms    15.00us   751.00us   480.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1101.45ms  1101.34ms     6.00us    68.00us    36.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
-total-              -    21.09ms    19.98ms    11.00us   719.00us   375.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
VOLUME            2178   995.61ms   995.50ms     6.00us    49.00us    50.00us        0ms        0ms     1.00us        0ms        0ms        0ms        0ms
-total-              -    26.80ms    25.16ms    12.00us  1206.00us   423.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1107.57ms  1107.45ms     4.00us    69.00us    45.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
-total-              -    22.77ms    21.48ms    13.00us   893.00us   376.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1486.07ms  1485.96ms     6.00us    58.00us    51.00us        0ms        0ms     1.00us        0ms        0ms        0ms        0ms
-total-              -    27.85ms    26.33ms    19.00us  1199.00us   299.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1439.01ms  1438.94ms     3.00us    52.00us     9.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms

• Multiple volumes on multiple storage nodes are impacted, without a pattern
• SRAM dumps, port resets, threshold reached, and or link break errors are logged intermittently on multiple storage nodes

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_thread_15: scsitarget.fcp.dump:debug]: FCP target SRAM dump generated for adapter 10a, fct_tpd_check_hung_commands: Command termination hung. cmd:0xfffff817e09cdb70 (state=0xa, flags=0x2,ctio_sent=1/1, RecvExAddr=0x11e620, OX_ID=0x4439, RX_ID=0xffff, SID=0x2729e2)

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_thread_15: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 10a.

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO TPD cmd alloc threshold reached handle:7 taskflags:128 Active commands:3891 threshold:3891

• Clients have MPIO configured properly with round robin policy and all optimized IO paths are active (two paths per Fabric)
• Low port discards increment on the switch and SFP power receive/transmit is not at low thresholds