Skip to main content
NetApp Knowledge Base

StorageGRID upgrade to 11.4 hangs at Rebooting stage

Views:
655
Visibility:
Public
Votes:
0
Category:
storagegrid-webscale
Specialty:
sgrid
Last Updated:

Applies to

  • NetApp StorageGRID 11.4
  • Pre-GRID Environment (PGE) 3.4
  • NetApp StorageGRID Appliance SG6060

Issue

  • During upgrade to StorageGRID 11.4 the installer hangs at "Rebooting" at around 75% of the node upgrade progress bar for more than 30-45 minutes.
  • The node is unreachable on the network, via BMC (Baseboard Management Contoller) console shows it's booted up to the base-os (this has a green prompt after login).
  • Inside base-os network interfaces are missing pre-configured/expected settings, and not all expected interfaces are listed in ifconfig output.
  • Listing network interfaces with ifconfig -a does list all network interfaces.
  • Listing devices in /dev/mapper only shows the control device:
     
    root@SG:~ # ls -al /dev/mapper/
    total 0
    drwxr-xr-x  2 root root     460 Sep  4 16:34 .
    drwxr-xr-x 17 root root    5420 Sep  4 16:34 ..
    crw-------  1 root root 10, 236 Sep  4 16:32 control
  • The /var/log/daemon.log log file in the base-os shows messages similar to the following repeating indefinitely:
Aug 21 12:53:13 localhost config-sga[2118]: [2020-08-21T12:53:13+00:00 CFGD] Disks are missing from /dev/mapper. Waiting 5 seconds.
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-PGE-Backup is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-SG-OS is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-00 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-01 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-02 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-03 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-04 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-05 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-06 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-07 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-08 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-09 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-10 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-11 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-12 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-13 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-14 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-15 is missing from /dev/mapper
  • And (less frequently) intermixed with:
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] Disks are missing from /dev/mapper. Restarting multipath-tools.
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm -r /dev/disk/by-label/* /dev/disk/by-uuid/*
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm: cannot remove '/dev/disk/by-label/*': No such file or directory
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm: cannot remove '/dev/disk/by-uuid/*': No such file or directory
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] dmsetup remove_all
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] service multipath-tools restart
Aug 21 12:53:18 localhost multipathd[10079]: exit (signal)
Aug 21 12:53:18 localhost systemd[1]: Stopping Device-Mapper Multipath Device Controller...
Aug 21 12:53:18 localhost multipathd[10079]: --------shut down-------
Aug 21 12:53:18 localhost systemd[1]: multipathd.service: Succeeded.
Aug 21 12:53:18 localhost systemd[1]: Stopped Device-Mapper Multipath Device Controller.
Aug 21 12:53:18 localhost systemd[1]: Starting Device-Mapper Multipath Device Controller...
Aug 21 12:53:18 localhost multipathd[11121]: --------start up--------
Aug 21 12:53:18 localhost multipathd[11121]: read /etc/multipath.conf
Aug 21 12:53:18 localhost multipathd[11121]: path checkers start up
Aug 21 12:53:18 localhost systemd[1]: Started Device-Mapper Multipath Device Controller.
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] Done restarting multipath-tools. Checking again in 60 seconds

 

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

Scan to view the article on your device