Skip to main content
NetApp Knowledge Base

Both admin and gateway nodes rebooted simultaneously during 11.8.0 upgrade and caused an outage

Views:
84
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

NetApp StorageGRID 11.7.0.X to 11.8.0

Issue

  • While upgrading StorageGRID firmware from 11.7.0.X to 11.8.0, both admin and gateway nodes rebooted simultaneously to apply the upgrade, resulting in an outage.
  • Admin node install.log:

[2024-03-20T23:05:30+00:00 INSG] Received SIGTERM; attempting to stop StorageGRID services
[2024-03-20T23:05:30+00:00 INSG]
[2024-03-20T23:05:30+00:00 INSG] *************************
[2024-03-20T23:05:30+00:00 INSG] *** Stopping Services ***
[2024-03-20T23:05:30+00:00 INSG] *************************
[2024-03-20T23:05:30+00:00 INSG]
[2024-03-20T23:05:30+00:00 INSG] [root@<admin_node>:/root] >>> rm -f /var/local/no-reboot-triggered
[2024-03-20T23:05:30+00:00 INSG] [root@<admin_node>:/root] >>> service keepalived stop
[2024-03-20T23:05:30+00:00 INSG] Stopping keepalived: keepalived.
[2024-03-20T23:05:30+00:00 INSG] [root@<admin_node>:/root] >>> ruby /usr/local/lib/site_ruby/bycast/servermanager/stopall.rb --verbose --timeout=720
[2024-03-20T23:05:30+00:00 INSG] remove all error states
[2024-03-20T23:05:31+00:00 INSG] Sent request to add storagegrid_administratively_down metric.
[2024-03-20T23:05:35+00:00 INSG] stopping all services
[2024-03-20T23:05:36+00:00 INSG] stopping acct-tunnel
[2024-03-20T23:05:36+00:00 INSG] stopping alertmanager
[2024-03-20T23:05:36+00:00 INSG] stopping ams
[2024-03-20T23:05:36+00:00 INSG] stopping attrDownPurge
[2024-03-20T23:05:36+00:00 INSG] stopping attrDownSamp1

  • Gateway node install.log:

[2024-03-20T23:04:53+00:00 INSG] Received SIGTERM; attempting to stop StorageGRID services
[2024-03-20T23:04:53+00:00 INSG]
[2024-03-20T23:04:53+00:00 INSG] *************************
[2024-03-20T23:04:53+00:00 INSG] *** Stopping Services ***
[2024-03-20T23:04:53+00:00 INSG] *************************
[2024-03-20T23:04:53+00:00 INSG]
[2024-03-20T23:04:53+00:00 INSG] [root@<gateway_node>:/root] >>> rm -f /var/local/no-reboot-triggered
[2024-03-20T23:04:53+00:00 INSG] [root@<gateway_node>:/root] >>> service keepalived stop
[2024-03-20T23:04:53+00:00 INSG] Stopping keepalived: keepalived.
[2024-03-20T23:04:53+00:00 INSG] [root@<gateway_node>:/root] >>> ruby /usr/local/lib/site_ruby/bycast/servermanager/stopall.rb --verbose --timeout=720
[2024-03-20T23:04:54+00:00 INSG] remove all error states
[2024-03-20T23:04:54+00:00 INSG] Sent request to add storagegrid_administratively_down metric.
[2024-03-20T23:04:55+00:00 INSG] stopping all services
[2024-03-20T23:04:56+00:00 INSG] stopping acct-tunnel
[2024-03-20T23:04:56+00:00 INSG] stopping dynip
[2024-03-20T23:04:56+00:00 INSG] stopping jaeger-agent
[2024-03-20T23:04:56+00:00 INSG] stopping miscd

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.