StorageGRID upgrade to 11.4 hangs at Rebooting stage

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 864

Visibility:: Public

Votes:: 0

Category:: storagegrid-webscale

Specialty:: sgrid

Last Updated:

Applies to

NetApp StorageGRID 11.4
Pre-GRID Environment (PGE) 3.4
NetApp StorageGRID Appliance SG6060

Issue

During upgrade to StorageGRID 11.4 the installer hangs at "Rebooting" at around 75% of the node upgrade progress bar for more than 30-45 minutes.
The node is unreachable on the network, via BMC (Baseboard Management Contoller) console shows it's booted up to the base-os (this has a green prompt after login).
Inside base-os network interfaces are missing pre-configured/expected settings, and not all expected interfaces are listed in ifconfig output.
Listing network interfaces with ifconfig -a does list all network interfaces.
Listing devices in /dev/mapper only shows the control device:

root@SG:~ # ls -al /dev/mapper/ total 0 drwxr-xr-x 2 root root 460 Sep 4 16:34 . drwxr-xr-x 17 root root 5420 Sep 4 16:34 .. crw------- 1 root root 10, 236 Sep 4 16:32 control
The /var/log/daemon.log log file in the base-os shows messages similar to the following repeating indefinitely:

Aug 21 12:53:13 localhost config-sga[2118]: [2020-08-21T12:53:13+00:00 CFGD] Disks are missing from /dev/mapper. Waiting 5 seconds.

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-PGE-Backup is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-SG-OS is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-00 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-01 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-02 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-03 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-04 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-05 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-06 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-07 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-08 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-09 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-10 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-11 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-12 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-13 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-14 is missing from /dev/mapper

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-15 is missing from /dev/mapper

And (less frequently) intermixed with:

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] Disks are missing from /dev/mapper. Restarting multipath-tools.

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm -r /dev/disk/by-label/* /dev/disk/by-uuid/*

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm: cannot remove '/dev/disk/by-label/*': No such file or directory

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm: cannot remove '/dev/disk/by-uuid/*': No such file or directory

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] dmsetup remove_all

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] service multipath-tools restart

Aug 21 12:53:18 localhost multipathd[10079]: exit (signal)

Aug 21 12:53:18 localhost systemd[1]: Stopping Device-Mapper Multipath Device Controller...

Aug 21 12:53:18 localhost multipathd[10079]: --------shut down-------

Aug 21 12:53:18 localhost systemd[1]: multipathd.service: Succeeded.

Aug 21 12:53:18 localhost systemd[1]: Stopped Device-Mapper Multipath Device Controller.

Aug 21 12:53:18 localhost systemd[1]: Starting Device-Mapper Multipath Device Controller...

Aug 21 12:53:18 localhost multipathd[11121]: --------start up--------

Aug 21 12:53:18 localhost multipathd[11121]: read /etc/multipath.conf

Aug 21 12:53:18 localhost multipathd[11121]: path checkers start up

Aug 21 12:53:18 localhost systemd[1]: Started Device-Mapper Multipath Device Controller.

Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] Done restarting multipath-tools. Checking again in 60 seconds