FCP SRAM DUMP and IO WQE failure with Ext_Status 0x16 caused by same host SID
Applies to
- ONTAP 9.x
- FC
Issue
EMS
Logs reportsIO WQE failure
withExt_Status 0x16.
- Extended status 0x16 further indicates that Host initiator has sent an abort to clear the current command queue.
Sun May 14 23:05:37 +0530 [NetApp: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:2b IO WQE failure, Handle 0x1, Type 8, S_ID: 61xx0, VPI: 276, OX_ID: 1B28, Status 0x3 Ext_Status 0x16
Sun May 14 23:05:50 +0530 [NetApp: fct_tpd_work_thread: ems.engine.suppressed:debug]: Event 'fcp.io.status' suppressed 1 times in last 5 seconds.
Sun May 14 23:05:50 +0530 [NetApp: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:3b IO WQE failure, Handle 0x3, Type 8, S_ID: 61xx0, VPI: 276, OX_ID: 2588, Status 0x3 Ext_Status 0x16
Sun May 14 23:06:05 +0530 [NetApp: fct_tpd_work_thread: ems.engine.suppressed:debug]: Event 'fcp.io.status' suppressed 2 times in last 15 seconds.
EMS
log on storage reportsFCP SRAMP Dump a
nd adapters being reset:
Sun Dec 03 17:08:54 0000 [NetApp: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:4b IO WQE failure, Handle 0x1, Type 8, S_ID: 0x61xx0, VPI: 275, OX_ID: 5FBB, Status 0x3 Ext_Status 0x16
Sun Dec 03 17:09:16 0000 [NetApp: fct_tpd_thread_1: scsitarget.fcp.dump:debug]: FCP target SRAM dump generated for adapter 4b, FW Initiated Dump
Sun Dec 03 17:09:16 0000 [NetApp: fct_tpd_thread_1: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 4b. The adapter will be automatically reset to clear the status:0x87800000, status1:0x52004c62, status2:0x610102, DIP:1, RN:1, RDY:1, Dump owner:1 condition.
Sun Dec 03 17:09:16 0000 [NetApp: fct_tpd_thread_0: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 4a. The adapter will be automatically reset to clear the status:0x87800000, status1:0x52004c62, status2:0x610102, DIP:1, RN:1, RDY:1, Dump owner:1 condition.
Sun Dec 03 17:09:16 0000 [NetApp: fct_tpd_thread_0: fcp.io.status:debug]: STIO Adapter 4a resetting with 28 ITNs and 10 commands to drain
Sun Dec 03 17:09:16 0000 [NetApp: fct_tpd_thread_0: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 4a.
Sun Dec 03 17:09:17 0000 [NetApp: fct_tpd_thread_1: scsitarget.hwpfct.dump.saved:notice]: A dump for adapter 4b was stored in /etc/log/fctsli_4b_20231203_170916/fct_fw_4b.dmp.gz.
Sun Dec 03 17:09:17 0000 [NetApp: fct_tpd_thread_1: callhome.fcp.sram.dump:error]: Call home for FCP SRAM DUMP.
Sun Dec 03 17:09:17 0000 [NetApp: fct_tpd_thread_1: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 4b.
Dec 03 17:09:18 0000 [NetApp: fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 4a.
Sun Dec 03 17:09:18 0000 [NetApp: fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 4b.
- STIO hung cmd event with
state=10
andstate =5
reported inEMS
for the sameSID 0x61xx0
Sun May 14 23:13:42 +0530 [NetApp: fct_tpd_thread_1: fcp.io.status:debug]: STIO Adapter:2b, found hung cmd:0xfffff8188234ab78(state=10, flags=0x2, ctio_sent=2/3,RecvExAddr=0x2ae5, OX_ID=0x1e66, RX_ID=0xffff,SID=0x61xx0, Cmd[28], req_q_free:0)
Wed Dec 20 21:36:21 +0000 [NetApp: fct_tpd_thread_3: fcp.io.status:debug]: STIO Adapter:4d, found hung cmd:0xfffff81e0a7d1038(state=5, flags=0x0, ctio_sent=1/5,RecvExAddr=0x271f, OX_ID=0x5ee9, RX_ID=0xffff,SID=0x61xx0, Cmd[2A], req_q_free:0)
- Note: state=5: DATAOUT_WAIT This indicates that the FC target is waiting for something to come back from the host after accepting the write request; however, nothing came back within the expected timeout value.
- Rx and Tx power are in optimal range on the storage ports.
- Check for any physical layer issues between host and switch like sfp, cable, patch panel.