ESXi hosts frequent disconnection from iSCSI Datastore
Applies to
- VMware ESXi
- VMware vSphere
- iSCSI Storage
- VMFS
Issue
- iSCSI VMFS Datastore goes into an inaccessible state
- On the FILER, check the vserver and igroup info (notice that IQN's are not logged in which should correspond to the problematic ESXi hosts)
<ONTAP_Server>::> igroup show -vserver <vserver_name> -igroup <igroup_name>
Vserver Name: <vserver_name>
Igroup Name: <igroup_name>
Protocol: iscsi
OS Type: vmware
Portset Binding Igroup: -
Igroup UUID: <Igroup_UUID>
ALUA: true
Initiators: iqn.1998-01.com.vmware:<host> (logged in)
iqn.1998-01.com.vmware:<host> (logged in)
iqn.1998-01.com.vmware:<host> (logged in)
iqn.1998-01.com.vmware:<host> (not logged in)
iqn.1998-01.com.vmware:<host> (not logged in)
iqn.1998-01.com.vmware:<host> (not logged in)
- 2. Review v
mkernel.logandvobd.log- here it's observed that the iSCSI connection is stopped
2020-03-31T07:12:05.671Z: [iscsiCorrelator] 82558981242us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba64:C2:T0
2020-03-31T07:12:05.671Z: [iscsiCorrelator] 82558981430us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba64:C0:T0
- 3. Just before the iSCSI connection is stopped, it's marked as OFFLINE
2020-03-31T07:12:05.670Z cpu39:66402)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba64:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2020-03-31T07:12:05.670Z cpu39:66402)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000003 TARGET: iqn.1992-08.com.netapp:sn.<device_serial_number>:vs.12 TPGT: 41a TSIH: 0]
2020-03-31T07:12:05.670Z cpu39:66402)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: <ip_address>:<port> R: <ip_address>:<port>]
- 4. Review ONTAP Logs:
Here it's observed that the IP addresses of the IQN's used on a Windows server and an ESXi host is the same.
Wed Apr 01 06:34:44 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:<Windows_host> at IP addr 10.10.10.10
Wed Apr 01 06:33:57 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1998-01.com.vmware:<ESXi_host> at IP addr 10.10.10.10
Wed Apr 01 05:37:42 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:<Windows_host> at IP addr 11.11.11.11
Wed Apr 01 05:37:26 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1998-01.com.vmware:<ESXi_host> at IP addr 11.11.11.11
Wed Apr 01 06:37:59 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:<Windows_host> at IP addr 12.12.12.12
Wed Apr 01 06:37:40 CEST [<ONTAP_SERVER>: iswt_admin_thread: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1998-01.com.vmware:<ESXi_host> at IP addr 12.12.12.12