ESXi hosts frequent disconnection from iSCSI Datastore

Applies to

• VMware ESXi
• VMware vSphere
• iSCSI Storage
• VMFS

Issue

• iSCSI VMFS Datastore goes into an inaccessible state
• On the FILER, check the vserver and igroup info (notice that IQN's are not logged in which should correspond to the problematic ESXi hosts)

 <ONTAP_Server>::> igroup show -vserver <vserver_name> -igroup <igroup_name>
          Vserver Name: <vserver_name>
Igroup Name: <igroup_name>

  Protocol: iscsi
               OS Type: vmware
Portset Binding Igroup: -
           Igroup UUID: <Igroup_UUID>
                  ALUA: true
Initiators: iqn.1998-01.com.vmware:<host> (logged in)
iqn.1998-01.com.vmware:<host> (logged in)
iqn.1998-01.com.vmware:<host> (logged in)
iqn.1998-01.com.vmware:<host> (not logged in)
iqn.1998-01.com.vmware:<host> (not logged in)
iqn.1998-01.com.vmware:<host> (not logged in)

 

• 2. Review vmkernel.log and vobd.log - here it's observed that the iSCSI connection is stopped

2020-03-31T07:12:05.671Z: [iscsiCorrelator] 82558981242us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba64:C2:T0
2020-03-31T07:12:05.671Z: [iscsiCorrelator] 82558981430us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba64:C0:T0

• 3. Just before the iSCSI connection is stopped, it's marked as OFFLINE

2020-03-31T07:12:05.670Z cpu39:66402)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba64:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2020-03-31T07:12:05.670Z cpu39:66402)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000003 TARGET: iqn.1992-08.com.netapp:sn.<device_serial_number>:vs.12 TPGT: 41a TSIH: 0]
2020-03-31T07:12:05.670Z cpu39:66402)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: <ip_address>:<port> R: <ip_address>:<port>]

 

• 4. Review ONTAP Logs: 

Here it's observed that the IP addresses of the IQN's used on a Windows server and an ESXi host is the same. 

Wed Apr 01 06:34:44 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:<Windows_host> at IP addr 10.10.10.10
Wed Apr 01 06:33:57 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1998-01.com.vmware:<ESXi_host> at IP addr 10.10.10.10

Wed Apr 01 05:37:42 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:<Windows_host> at IP addr 11.11.11.11
Wed Apr 01 05:37:26 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1998-01.com.vmware:<ESXi_host> at IP addr 11.11.11.11

Wed Apr 01 06:37:59 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:<Windows_host> at IP addr 12.12.12.12
Wed Apr 01 06:37:40 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1998-01.com.vmware:<ESXi_host> at IP addr 12.12.12.12