Skip to main content
NetApp Knowledge Base

StorageGRID upgrade to 11.4 hangs at Rebooting stage

Views:
837
Visibility:
Public
Votes:
0
Category:
storagegrid-webscale
Specialty:
sgrid
Last Updated:

Applies to

  • NetApp StorageGRID 11.4
  • Pre-GRID Environment (PGE) 3.4
  • NetApp StorageGRID Appliance SG6060

Issue

  • During upgrade to StorageGRID 11.4 the installer hangs at "Rebooting" at around 75% of the node upgrade progress bar for more than 30-45 minutes.
  • The node is unreachable on the network, via BMC (Baseboard Management Contoller) console shows it's booted up to the base-os (this has a green prompt after login).
  • Inside base-os network interfaces are missing pre-configured/expected settings, and not all expected interfaces are listed in ifconfig output.
  • Listing network interfaces with ifconfig -a does list all network interfaces.
  • Listing devices in /dev/mapper only shows the control device:
     
    root@SG:~ # ls -al /dev/mapper/
    total 0
    drwxr-xr-x  2 root root     460 Sep  4 16:34 .
    drwxr-xr-x 17 root root    5420 Sep  4 16:34 ..
    crw-------  1 root root 10, 236 Sep  4 16:32 control
  • The /var/log/daemon.log log file in the base-os shows messages similar to the following repeating indefinitely:
Aug 21 12:53:13 localhost config-sga[2118]: [2020-08-21T12:53:13+00:00 CFGD] Disks are missing from /dev/mapper. Waiting 5 seconds.
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-PGE-Backup is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-SG-OS is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-00 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-01 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-02 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-03 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-04 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-05 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-06 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-07 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-08 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-09 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-10 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-11 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-12 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-13 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-14 is missing from /dev/mapper
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] StorageGRID-obj-15 is missing from /dev/mapper
  • And (less frequently) intermixed with:
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] Disks are missing from /dev/mapper. Restarting multipath-tools.
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm -r /dev/disk/by-label/* /dev/disk/by-uuid/*
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm: cannot remove '/dev/disk/by-label/*': No such file or directory
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] rm: cannot remove '/dev/disk/by-uuid/*': No such file or directory
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] dmsetup remove_all
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] service multipath-tools restart
Aug 21 12:53:18 localhost multipathd[10079]: exit (signal)
Aug 21 12:53:18 localhost systemd[1]: Stopping Device-Mapper Multipath Device Controller...
Aug 21 12:53:18 localhost multipathd[10079]: --------shut down-------
Aug 21 12:53:18 localhost systemd[1]: multipathd.service: Succeeded.
Aug 21 12:53:18 localhost systemd[1]: Stopped Device-Mapper Multipath Device Controller.
Aug 21 12:53:18 localhost systemd[1]: Starting Device-Mapper Multipath Device Controller...
Aug 21 12:53:18 localhost multipathd[11121]: --------start up--------
Aug 21 12:53:18 localhost multipathd[11121]: read /etc/multipath.conf
Aug 21 12:53:18 localhost multipathd[11121]: path checkers start up
Aug 21 12:53:18 localhost systemd[1]: Started Device-Mapper Multipath Device Controller.
Aug 21 12:53:18 localhost config-sga[2118]: [2020-08-21T12:53:18+00:00 CFGD] Done restarting multipath-tools. Checking again in 60 seconds

 

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.