Lost access to volume due to connectivity issues
Applies to
- FC LUN
- Brocade switch
- Ontap 9.x
Issue
- FC LUN lost its access on host end due to connectivity issues.
IO WQE
errors were observed on storge side as below on both the nodes.
Tue May 09 09:00:03 -0400 [NetApp-01: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:5b IO WQE failure, Handle 0x1, Type 8, S_ID: 401xx, VPI: 275, OX_ID: 63F, Status 0x3 Ext_Status 0x16
Tue May 09 09:00:33 -0400 [NetApp-01: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:5b IO WQE failure, Handle 0x1, Type 8, S_ID: 401xx, VPI: 275, OX_ID: 4E6, Status 0x3 Ext_Status 0x16
Tue May 09 09:00:52 -0400 [NetApp-02: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:5b IO WQE failure, Handle 0x1, Type 8, S_ID: 401xx, VPI: 275, OX_ID: 58E, Status 0x3 Ext_Status 0x16
Tue May 09 09:08:16 -0400 [NetApp-02: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:5b IO WQE failure, Handle 0x1, Type 8, S_ID: 401xx, VPI: 275, OX_ID: 219, Status 0x3 Ext_Status 0x1d
Tue May 09 09:29:05 -0400 [NetApp-02: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:5b IO WQE failure, Handle 0x1, Type 8, S_ID: 401xx, VPI: 275, OX_ID: 71, Status 0x3 Ext_Status 0x2
C3 timeout Tx
errors were observed on the storage connected ports where WQE errors were reported along with the ISL ports on switch side:
Index Port Address Media Speed State Proto
==================================================
8 8 020800 id N32 Online FC F-Port 1 N Port + 1 NPIV public
9 9 020900 id N32 Online FC F-Port 1 N Port + 1 NPIV public
46 46 022e00 id 16G Online FC LE E-Port 10:00:88:94:71:9d:xx:xx "SW-XX"
47 47 022f00 id N16 Online FC LE E-Port 10:00:88:94:71:9d:yy:yy "SW-XX" (downstream)
/fabos/cliexec/porterrshow:
frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs uncor
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err err
8: 1.0g 185.7m 0 1 1 0 0 0 0 48.4k 4 0 4 0 0 0 8.5k 0 3
9: 4.0g 1.3g 0 5 5 0 0 0 0 64.4k 4 0 4 0 0 0 17.4k 0 5
46: 2.1g 157.7m 0 0 0 0 0 10.5k 0 27.4k 121 1 0 0 0 26.4k 0 20.1k 0
47: 3.7g 348.5m 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0
Frame timeout detected
errors reported under errrdump indicating the frame could not be sent to the ISL port ( in this case Tx port 46)
2023/05/09-11:44:52, [AN-1014], 155073, FID 128, INFO, Switch_name, Frame timeout detected, tx port 46 rx port 8, sid 20801, did 40000, timestamp 2023-05-09 11:44:52
2023/05/09-11:44:53, [AN-1014], 155093, FID 128, INFO, Switch_name, Frame timeout detected, tx port 46 rx port 8, sid 20801, did 40000, timestamp 2023-05-09 11:44:53