ONTAP SAN all FC ports going down with fcp.sram.dump
Applies to
- ONTAP
- Fiber Channel Protocol ( FCP )
Issue
- When making fabric changes multiple storage FC ports reset with SRAM dump
- Scanning LUN with multiple VMware hosts FC ports reset with SRAM dump
- Fabric state change notification (SCN) FC ports reset with SRAM dump
- More than one Storage vMotion FC ports reset with SRAM dump
- Adding or removing VMware Datastores cause FC port reset with SRAM dump
- Resets to clear the status
0x87800000 status1 0xbcbf status2 0x4 - Frame timeouts always involving an ISL as receive ( Rx) or transmit (Tx)/ source ( SID) or destination (DID)
Example:
ONTAP:
event log show -event *fct* -time < 10m
fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:2c IO WQE failure, Handle 0x2, Type 8, S_ID: 123, VPI: 257, OX_ID: 89B, Status 0x3 Ext_Status 0x1d
fcp.io.status:debug]: STIO Adapter:2c, found hung cmd:(state=10, flags=0x2, ctio_sent=2/2,RecvExAddr=, OX_ID=0x1123, RX_ID=abc,SID=0x123, Cmd[89], req_q_free:0)
fct_tpd_thread_2: scsitarget.fcp.dump:debug]: FCP target SRAM dump generated for adapter 2c, fct_tpd_check_hung_commands: Command termination hung. cmd:0x123 (state=0xa, flags=0x2,ctio_sent=2/2, RecvExAddr=0123, OX_ID=0x123, RX_ID=123, SID=0x123)
fct_tpd_thread_0: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 2a. The adapter will be automatically reset to clear the status:0x87800000, status1:0xa00088d6, status2:0xffffffff, DIP:1, RN:1, RDY:1, Dump owner:2 condition.
fct_tpd_thread_1: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 2b. The adapter will be automatically reset to clear the status:0x87800000, status1:0xa00088d6, status2:0xffffffff, DIP:1, RN:1, RDY:1, Dump owner:2 condition.
fct_tpd_thread_3: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 2d. The adapter will be automatically reset to clear the status:0x87800000, status1:0xa00088d6, status2:0xffffffff, DIP:1, RN:1, RDY:1, Dump owner:2 condition.
fct_tpd_thread_3: fcp.io.status:debug]: STIO Adapter 2d resetting with 35 ITNs and 358 commands to drain
fct_tpd_thread_3: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 2d. < ASIC B
fct_tpd_thread_1: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 2b. < ASIC B
fct_tpd_thread_2: scsitarget.hwpfct.errorReset:notice]: An error was encountered in the FC target driver on Fibre Channel target adapter 2c. The adapter will be automatically reset to clear the status:0x87800000, status1:0xa00088d6, status2:0xffffffff, DIP:1, RN:1, RDY:1, Dump owner:2 condition.
fct_tpd_thread_2: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 2c. < ASIC A
fct_tpd_thread_0: scsitarget.fct.reset:notice]: Resetting Fibre Channel target adapter 2a. < ASIC A
fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 2a.
fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 2b.
fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 2c.
fct_tpd_work_thread_0: scsitarget.hwpfct.linkUp:notice]: Link up on Fibre Channel target adapter 2d.Brocade:
errdump -aINFO, 6505, Frame timeout detected, tx port 1 rx port 5, sid 12345, did 12345, timestamp 2024-08-13 17:11:00........