Skip to main content
NetApp Knowledge Base

Lost access to volume events due to faulty inter-switch link

  1. Last updated
  2. Save as PDF
Views:
472
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
san
Last Updated:

Applies to

  • ONTAP 9
  • SnapMirror active sync (SMas)
  • Brocade FC switch
  • Two sites connected with Inter-switch link (ISL)
  • VMware ESXi

Issue

  • VMware shows intermittent lost access to volume events:
Lost access to volume 63864940-ca00f732-af03-0025b511000e (example_ds1) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
Successfully restored access to volume 63864940-ca00f732-af03-0025b511000e (example_ds1) following connectivity issues.
  • EMS logs fibre channel issues which affected only hosts at opposite site of the cluster, as seen by the logged S_ID:

[cluster1_node1: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:2b IO WQE failure, Handle 0x1, Type 8, S_ID: 16210D, VPI: 130, OX_ID: 46D8, Status 0x3 Ext_Status 0x16
[cluster1_node1: fct_tpd_thread_1: fcp.io.status:debug]: STIO Adapter:2b, found hung cmd:0xfffff8042aea04b0(state=5, flags=0x0, ctio_sent=1/1,RecvExAddr=0x14c7, OX_ID=0x4947, RX_ID=0xffff,SID=0x16210d, Cmd[2A], req_q_free:0)   
[cluster1_node2: fct_tpd_thread_1: fcp.io.status:debug]: STIO Adapter:2b, found hung cmd:0xfffff8042af79d28(state=5, flags=0x0, ctio_sent=1/1,RecvExAddr=0x16b0, OX_ID=0x4fb7, RX_ID=0xffff,SID=0x16190b, Cmd[2A], req_q_free:0)
[cluster2_node1: fct_tpd_thread_1: fcp.io.status:debug]: STIO Adapter:2b, found hung cmd:0xfffff804254b0478(state=5, flags=0x0, ctio_sent=1/1,RecvExAddr=0x1666, OX_ID=0x44ce, RX_ID=0xffff,SID=0xc2109, Cmd[2A], req_q_free:0)
[cluster2_node1: fct_tpd_thread_1: fcp.io.status:debug]: STIO Adapter:2b, found hung cmd:0xfffff80425401d10(state=5, flags=0x0, ctio_sent=1/1,RecvExAddr=0x1568, OX_ID=0x4291, RX_ID=0xffff,SID=0xc2109, Cmd[2A], req_q_free:0)
[cluster2_node2: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:2b IO WQE failure, Handle 0x1, Type 8, S_ID: 16210D, VPI: 130, OX_ID: 4273, Status 0x3 Ext_Status 0x11

  • One switch shows an ISL has low receive power:
RX Power: -6.1 dBm (245.0uW)
TX Power: 0.4 dBm (1093.9 uW)
  • Power is good on the other side of the ISL:
RX Power:    -2.8    dBm (523.9uW)
TX Power:    -0.2    dBm (949.3 uW)
  • Production is impacted for VMs on hosts accessing datastores on the other site, over the ISL

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.