SnapCenter leaves dead paths in VMware vCenter after unmapping disks
Applies to
- SnapCenter 4.X
- SnapCenter Plugin for SQL (SCSQL)
- ESXi 7.0.3, 21313628
Issue
- After SCSQL unmaps disks from a cloning/verification/restore operation in vCenter 7.0.3, 21313628 dead paths remain
- In the vobd.log file from the vCenter log bundle the following can be seen:
2023-06-10T05:31:26.601Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.add] Add path: vmhba3:C0:T0:L50 2023-06-10T05:31:26.602Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.add] Add path: vmhba3:C0:T1:L50 2023-06-10T05:31:26.605Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba3:C0:T1:L50 changed state from dead 2023-06-10T05:31:26.608Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba3:C0:T0:L50 changed state from dead 2023-06-10T05:36:30.408Z: [APDCorrelator] hostname: [vob.storage.apd.start] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c52] has entered the All Paths Down state. 2023-06-10T05:36:30.408Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba3:C0:T1:L44 changed state from on 2023-06-10T05:36:30.409Z: [APDCorrelator] hostname: [esx.problem.storage.apd.start] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c52] has entered the All Paths Down state. 2023-06-10T05:36:30.412Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c52. Path vmhba3:C0:T1:L44 is down. Affected datastores: Unknown. 2023-06-10T05:36:30.412Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba2:C0:T1:L44 changed state from on 2023-06-10T05:36:30.414Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c52. Path vmhba2:C0:T1:L44 is down. Affected datastores: Unknown. 2023-06-10T05:36:30.414Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba3:C0:T0:L44 changed state from on 2023-06-10T05:36:30.416Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c52. Path vmhba3:C0:T0:L44 is down. Affected datastores: Unknown. 2023-06-10T05:36:30.416Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba2:C0:T0:L44 changed state from on 2023-06-10T05:36:30.419Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c52. Path vmhba2:C0:T0:L44 is down. Affected datastores: Unknown. 2023-06-10T05:36:35.082Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba3:C0:T1:L50 changed state from on 2023-06-10T05:36:35.082Z: [APDCorrelator] hostname: [vob.storage.apd.start] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c53] has entered the All Paths Down state. 2023-06-10T05:36:35.083Z: [APDCorrelator] hostname: [esx.problem.storage.apd.start] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c53] has entered the All Paths Down state. 2023-06-10T05:36:35.087Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c53. Path vmhba3:C0:T1:L50 is down. Affected datastores: Unknown. 2023-06-10T05:36:35.087Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba2:C0:T1:L50 changed state from on 2023-06-10T05:36:35.090Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c53. Path vmhba2:C0:T1:L50 is down. Affected datastores: Unknown. 2023-06-10T05:36:35.090Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba3:C0:T0:L50 changed state from on 2023-06-10T05:36:35.094Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c53. Path vmhba3:C0:T0:L50 is down. Affected datastores: Unknown. 2023-06-10T05:36:35.094Z: [scsiCorrelator] hostname: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba2:C0:T0:L50 changed state from on 2023-06-10T05:36:35.097Z: [scsiCorrelator] hostname: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.600a098038314a776b3f55666d684c53. Path vmhba2:C0:T0:L50 is down. Affected datastores: Unknown. 2023-06-10T05:38:50.411Z: [APDCorrelator] hostname: [vob.storage.apd.timeout] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c52] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed. 2023-06-10T05:38:50.412Z: [APDCorrelator] hostname: [esx.problem.storage.apd.timeout] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c52] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed. 2023-06-10T05:38:55.084Z: [APDCorrelator] hostname: [vob.storage.apd.timeout] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c53] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed. 2023-06-10T05:38:55.084Z: [APDCorrelator] hostname: [esx.problem.storage.apd.timeout] Device or filesystem with identifier [naa.600a098038314a776b3f55666d684c53] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.