Skip to main content
NetApp Knowledge Base

StorageGRID Upgrade to 11.4 gets stuck in "waiting for reboot" step or reboots but not fully comes back

Views:
119
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

  • StorageGRID upgrade to 11.4
  • Docker service on container nodes

Issue

Upgrade to 11.4:

  • gets stuck at the "waiting for reboot" step
  • or, a node reboots for some other reason and does not come back
    • services don't start
    • node is "blue" in GRID Manager UI
    • normal ssh to port 22 does not connect, but can be reached via ssh to port 8022

 

  1. SSH into the base OS level on the node
    • ssh -p 8022
  2. Check /var/log/syslog for logs similar to the following:

Jan 13 20:06:25 localhost dockerd[784]: time="2020-01-13T20:06:25.225330454Z" level=error msg="failed connecting to containerd" error="failed to dial \"/var/run/docker/containerd/containerd.sock\": context deadline exceeded" module=libcontainerd
Jan 13 20:06:25 localhost dockerd[784]: time="2020-01-13T20:06:25.325657112Z" level=info msg="killing and restarting containerd" module=libcontainerd pid=825
Jan 13 20:06:25 localhost dockerd[784]: time="2020-01-13T20:06:25.426661368Z" level=error msg="containerd did not exit successfully" error="signal: killed" module=libcontainerd
Jan 13 20:06:26 localhost dockerd[784]: panic: runtime error: invalid memory address or nil pointer dereference
Jan 13 20:06:26 localhost dockerd[784]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x561adb97d4d0]
Jan 13 20:06:26 localhost dockerd[784]: goroutine 75 [running]:
Jan 13 20:06:26 localhost dockerd[784]: github.com/containerd/containerd.(*Client).Close(0x0, 0x0, 0x0)
Jan 13 20:06:26 localhost dockerd[784]: #011/build/docker.io-s7YiOg/docker.io-18.09.1+dfsg1/.gopath/src/github.com/containerd/containerd/client.go:536 +0x30
Jan 13 20:06:26 localhost dockerd[784]: github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc0008329c0, 0x561adcb3c660, 0xc00016c6c0)
Jan 13 20:06:26 localhost dockerd[784]: #011/build/docker.io-s7YiOg/docker.io-18.09.1+dfsg1/.gopath/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:321 +0x24f
Jan 13 20:06:26 localhost dockerd[784]: created by github.com/docker/docker/libcontainerd/supervisor.Start
Jan 13 20:06:26 localhost dockerd[784]: #011/build/docker.io-s7YiOg/docker.io-18.09.1+dfsg1/.gopath/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:90 +0x3b5

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.