StorageGRID Appliance gets stuck at rebooting stage during 11.4 upgrade
Applies to
- StorageGRID Appliance
- StorageGRID 11.4.0
Issue
- During upgrade to StorageGRID 11.4.0, the StorageGRID Appliance hangs at "Rebooting":
- The appliance stays at error code EP: Installation or communication with the E2800 controller has failed.
- Reboot the appliance from Installer, it stays at HO: The StorageGRID Appliance Installer is running.
- Review console log of the application during reboot, the appliance is unable to find volume StorageGRID-SG-OS-part1 after several retries to rebuild
multipath.conf:
[2021-07-12 03:06:14+00:00 SGA] WARNING -- Not all volumes are present in /dev/mapper CHECK MULTIPATH CONFIG
[2021-07-12 03:06:14+00:00 SGA] INFO -- Missing volume mappings; rewriting multipath.conf and recalculating in 15 seconds
...
[2021-07-12 03:06:16+00:00 SGA] INFO -- Wrote multipath config file.
[ 151.131520] scsi 6:0:0:7: Power-on or device reset occurred
[ 153.754420] scsi 7:0:0:7: Power-on or device reset occurred
[2021-07-12 03:06:25+00:00 SGA] INFO -- Flushing multipath devices
[2021-07-12 03:06:25+00:00 SGA] INFO -- Clearing dm devices
[2021-07-12 03:06:25+00:00 SGA] INFO -- Restarting multipath-tools daemon
[2021-07-12 03:06:40+00:00 SGA] INFO -- Reloading multipath devices
...
[2021-07-12 03:06:56+00:00 SGA] WARNING -- Not all volumes are present in /dev/mapper CHECK MULTIPATH CONFIG
[2021-07-12 03:06:56+00:00 SGA] INFO -- Missing volume mappings; rewriting multipath.conf and recalculating in 15 seconds
...
[2021-07-12 03:07:37+00:00 SGA] WARNING -- Not all volumes are present in /dev/mapper CHECK MULTIPATH CONFIG
[2021-07-12 03:07:37+00:00 SGA] INFO -- Missing volume mappings; rewriting multipath.conf and recalculating in 15 seconds
...
[2021-07-12 03:08:18+00:00 SGA] WARNING -- Not all volumes are present in /dev/mapper CHECK MULTIPATH CONFIG
[2021-07-12 03:08:18+00:00 SGA] INFO -- Missing volume mappings; rewriting multipath.conf and recalculating in 15 seconds
...
[2021-07-12 03:08:59+00:00 SGA] WARNING -- Not all volumes are present in /dev/mapper CHECK MULTIPATH CONFIG
[2021-07-12 03:08:59+00:00 SGA] ERROR -- Mappings do not exist!
[2021-07-12 03:08:59+00:00 SGA] /usr/local/bin/WaitForEOS.py returned non-zero exit code; forcing install
[2021-07-12 03:08:59+00:00 SGA] [root@StorageGRID-PGE:/root] >>> rm -f /tmp/maintenance.mode
[2021-07-12 03:08:59+00:00 SGA] Enable CFW download on E-series as we are in PGE mode.
...
[2021-07-12 03:08:59+00:00 SGA] [2021-07-12T03:08:59.775] - INFO -- Not loading synced data from SG OS LUN. Device /dev/mapper/StorageGRID-SG-OS-part1 does not exist
...
[2021-07-12 03:10:06+00:00 SGA] [root@StorageGRID-PGE:/usr/local/lib/pge-mgmt-api] >>> sleep 1
[2021-07-12 03:10:07+00:00 SGA] [root@StorageGRID-PGE:/usr/local/lib/pge-mgmt-api] >>> mount /dev/sda1 /mnt/pge-inac-part
[ 375.549105] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[2021-07-12 03:10:07+00:00 SGA] [root@StorageGRID-PGE:/usr/local/lib/pge-mgmt-api] >>> umount /mnt/pge-inac-part