Skip to main content
NetApp Knowledge Base

FCP SRAM DUMP and IO WQE failure with Ext_Status 0x16 caused by same host SID

  1. Last updated
  2. Save as PDF
Views:
958
Visibility:
Public
Votes:
0
Category:
fas-systems
Specialty:
san
Last Updated:

Applies to

  • ONTAP 9
  • FC

Issue

  • EMS Logs show IO WQE failure with Ext_Status 0x16 and 0x1d
    • Ext_Status 0x1d indicates a frame was rejected by the storage adapter because it received out-of-order data
    • Extended status 0x16 indicates that Host initiator has sent an abort to clear the current command queue

Sun May 14 23:05:37 +0530 [NetApp: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:2b IO WQE failure, Handle 0x1, Type 8, S_ID: 61xx0, VPI: 276, OX_ID: 1B28, Status 0x3 Ext_Status 0x16
Sun May 14 23:05:50 +0530 [NetApp: fct_tpd_work_thread: ems.engine.suppressed:debug]: Event 'fcp.io.status' suppressed 1 times in last 5 seconds.
Sun May 14 23:05:50 +0530 [NetApp: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:3b IO WQE failure, Handle 0x3, Type 8, S_ID: 61xx0, VPI: 276, OX_ID: 2588, Status 0x3 Ext_Status 0x16
Sun May 14 23:06:05 +0530 [NetApp: fct_tpd_work_thread: ems.engine.suppressed:debug]: Event 'fcp.io.status' suppressed 2 times in last 15 seconds.

  • EMS log on storage reports FCP SRAMP Dump and adapters being reset:

Sun Dec 03 17:08:54  0000 [NetApp: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:4b IO WQE failure, Handle 0x1, Type 8, S_ID: 0x61xx0, VPI: 275, OX_ID: 5FBB, Status 0x3 Ext_Status 0x16

Sun Dec 03 17:09:16  0000 [NetApp: fct_tpd_thread_1: scsitarget.fcp.dump:debug]: FCP target SRAM dump generated for adapter 4b, FW Initiated Dump
Sun Dec 03 17:09:16  0000 [NetApp: fct_tpd_thread_1: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 4b. The adapter will be automatically reset to clear the status:0x87800000, status1:0x52004c62, status2:0x610102, DIP:1, RN:1, RDY:1, Dump owner:1 condition.
Sun Dec 03 17:09:16  0000 [NetApp: fct_tpd_thread_0: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 4a. The adapter will be automatically reset to clear the status:0x87800000, status1:0x52004c62, status2:0x610102, DIP:1, RN:1, RDY:1, Dump owner:1 condition.
Sun Dec 03 17:09:16  0000 [NetApp: fct_tpd_thread_0: fcp.io.status:debug]: STIO Adapter 4a resetting with 28 ITNs and 10 commands to drain
Sun Dec 03 17:09:16  0000 [NetApp: fct_tpd_thread_0: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 4a.
Sun Dec 03 17:09:17  0000 [NetApp: fct_tpd_thread_1: scsitarget.hwpfct.dump.saved:notice]: A dump for adapter 4b was stored in /etc/log/fctsli_4b_20231203_170916/fct_fw_4b.dmp.gz.
Sun Dec 03 17:09:17  0000 [NetApp: fct_tpd_thread_1: callhome.fcp.sram.dump:error]: Call home for FCP SRAM DUMP.
Sun Dec 03 17:09:17  0000 [NetApp: fct_tpd_thread_1: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 4b.

Dec 03 17:09:18  0000 [NetApp: fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 4a.
Sun Dec 03 17:09:18  0000 [NetApp: fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 4b.
 

  • STIO hung cmd event with state=10  and state =5 reported in EMS for the same SID 0x61xx0 

     Sun May 14 23:13:42 +0530 [NetApp: fct_tpd_thread_1: fcp.io.status:debug]: STIO Adapter:2b, found hung cmd:0xfffff8188234ab78(state=10, flags=0x2, ctio_sent=2/3,RecvExAddr=0x2ae5, OX_ID=0x1e66, RX_ID=0xffff,SID=0x61xx0, Cmd[28], req_q_free:0)

Wed Dec 20 21:36:21 +0000 [NetApp: fct_tpd_thread_3: fcp.io.status:debug]: STIO Adapter:4d, found hung cmd:0xfffff81e0a7d1038(state=5, flags=0x0, ctio_sent=1/5,RecvExAddr=0x271f, OX_ID=0x5ee9, RX_ID=0xffff,SID=0x61xx0, Cmd[2A], req_q_free:0)

  • state=5: DATAOUT_WAIT  indicates that the FC target is waiting for something to come back from the host after accepting the write request; however, nothing came back within the expected timeout value.
  • Rx and Tx power are in optimal range on the storage ports.
  • Check for any physical layer issues between host and switch like sfp, cable, patch panel

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.