Skip to main content
NetApp Response to Russia-Ukraine Cyber Threat
In response to the recent rise in cyber threat due to the Russian-Ukraine crisis, NetApp is actively monitoring the global security intelligence and updating our cybersecurity measures. We follow U.S. Federal Government guidance and remain on high alert. Customers are encouraged to monitor the Cybersecurity and Infrastructure Security (CISA) website for new information as it develops and remain on high alert.
NetApp Knowledge Base

FCP slowness and SRAM dumps during higher workload

Views:
332
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
san
Last Updated:

Applies to

  • ONTAP 9
  • SAN
  • Fibre Channel (FC)

Issue

  • Slowness reported by users that have servers utilizing FCP LUNs
  • Scheduled jobs or planned workload is running in the environment
  • Large deployment of 30+ hosts, 10+ storage nodes
  • QOS volume latency show indicates network is highest contributor of latency
cluster::> qos statistics volume latency show -vserver vserver -volume volume
Workload            ID    Latency    Network    Cluster       Data       Disk    QoS Max    QoS Min      NVRAM      Cloud  FlexCache    SM Sync         VA
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total-              -    24.15ms    22.90ms    15.00us   751.00us   480.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1101.45ms  1101.34ms     6.00us    68.00us    36.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
-total-              -    21.09ms    19.98ms    11.00us   719.00us   375.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
VOLUME            2178   995.61ms   995.50ms     6.00us    49.00us    50.00us        0ms        0ms     1.00us        0ms        0ms        0ms        0ms
-total-              -    26.80ms    25.16ms    12.00us  1206.00us   423.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1107.57ms  1107.45ms     4.00us    69.00us    45.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
-total-              -    22.77ms    21.48ms    13.00us   893.00us   376.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1486.07ms  1485.96ms     6.00us    58.00us    51.00us        0ms        0ms     1.00us        0ms        0ms        0ms        0ms
-total-              -    27.85ms    26.33ms    19.00us  1199.00us   299.00us        0ms        0ms     3.00us        0ms        0ms        0ms        0ms
VOLUME            2178  1439.01ms  1438.94ms     3.00us    52.00us     9.00us        0ms        0ms     2.00us        0ms        0ms        0ms        0ms
  • Multiple volumes on multiple storage nodes are impacted, without a pattern
  • SRAM dumps, port resets, threshold reached, and or link break errors are logged intermittently on multiple storage nodes

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_thread_15: scsitarget.fcp.dump:debug]: FCP target SRAM dump generated for adapter 10a, fct_tpd_check_hung_commands: Command termination hung. cmd:0xfffff817e09cdb70 (state=0xa, flags=0x2,ctio_sent=1/1, RecvExAddr=0x11e620, OX_ID=0x4439, RX_ID=0xffff, SID=0x2729e2)

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_thread_15: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 10a.

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO TPD cmd alloc threshold reached handle:7 taskflags:128 Active commands:3891 threshold:3891

Sun Feb 21 XX:XX:XX -0500 [NODE: fct_tpd_work_thread_0: scsitarget.ispfct.linkBreak:error]: Link break detected on Fibre Channel target adapter 2a. Firmware status code status1 0x2, status2 0x7, and status4 0x0.

  • Clients have MPIO configured properly with round robin policy and all optimized IO paths are active (two paths per Fabric)
  • Low port discards increment on the switch and SFP power receive/transmit is not at low thresholds

 

Scan to view the article on your device
CUSTOMER EXCLUSIVE CONTENT

Registered NetApp customers get unlimited access to our dynamic Knowledge Base.

New authoritative content is published and updated each day by our team of experts.

Current Customer or Partner?

Sign In for unlimited access

New to NetApp?

Learn more about our award-winning Support