Skip to main content
NetApp Knowledge Base

Multiple storage nodes shutdown simultaneously

  1. Last updated
  2. Save as PDF
Views:
49
Visibility:
Public
Votes:
0
Category:
h-series
Specialty:
solidfire
Last Updated:

Applies to

  • H610S
  • Element OS 12.3.2.3

Issue

Multiple storage nodes shutdown simultaneously
 

Error log:
123 2023-10-14T07:31:20.206Z Error node 3 node-03 0 Yes 2023-10-14T08:14:23.042Z nodeOffline The SolidFire Application cannot communicate with Storage node having node ID 3.
124 2023-10-14T07:31:20.206Z Error node 4 node-04 0 Yes 2023-10-14T08:14:23.043Z nodeOffline The SolidFire Application cannot communicate with Storage node having node ID 4.

sf-master.info
2023-10-14T07:30:46.869482Z node-03 sfnetwd[18171]: [APP-5] [Net] 18171 sfnetwd sfnetwd/main.cpp:184:MonitorInterfaces|Shutting down on signal 15
2023-10-14T07:30:46.912309Z node-03 sfnetwd[18171]: [APP-5] [Process] 18176 signal_handler shared/Signal.cpp:216:ProcessSignals|Signal handler shutdown.
2023-10-14T07:30:46.912353Z node-03 sfnetwd[18171]: [APP-5] [Process] 18176 signal_handler shared/Signal.cpp:240:ProcessSignals|Signal handler thread exiting.
2023-10-14T07:30:46.912561Z node-03 sfnetwd[18171]: [APP-5] [Util] 18171 sfnetwd shared/LoopingThread.cpp:34:~LoopingThread|Destroying LoopingThread name=HangDetect
2023-10-14T07:30:46.919885Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/main.cpp:209:SigIntHandler|Service manager going down on signal: 15 from pid: 1 uid: 0
2023-10-14T07:30:46.920010Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:643:StopAllProcesses|Stopping all slice services...
2023-10-14T07:30:46.920105Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 20 PID: 25768
2023-10-14T07:30:46.920328Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:643:StopAllProcesses|Stopping all block services...
2023-10-14T07:30:46.920360Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 47 PID: 25756
2023-10-14T07:30:46.920390Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 51 PID: 25757
2023-10-14T07:30:46.920418Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 54 PID: 25758
2023-10-14T07:30:46.920449Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 57 PID: 25759
2023-10-14T07:30:46.920477Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 64 PID: 25760
2023-10-14T07:30:46.920515Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 23 PID: 25761
2023-10-14T07:30:46.920543Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 26 PID: 25762
2023-10-14T07:30:46.920572Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 32 PID: 25763
2023-10-14T07:30:46.920608Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 34 PID: 25764
2023-10-14T07:30:46.920650Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 39 PID: 2576
2023-10-14T07:30:46.920738Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 43 PID: 25766
2023-10-14T07:30:46.922675Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:643:StopAllProcesses|Stopping all fibre services...
2023-10-14T07:30:46.923569Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:643:StopAllProcesses|Stopping all master services...
2023-10-14T07:30:46.923618Z node-03 sfsvcmgr[18170]: [APP-5] [Process] 18180 signal_handler sfsvcmgr/sfsvcmgr.cpp:648:StopAllProcesses|  Stopping serviceID: 0 PID: 18533
2023-10-14T07:30:46.923995Z node-03 master-1[18533]: [APP-5] [Process] 19888 signal_handler serviceshared/ApplicationSimple.cpp:169:TermHandler|SIGTERM received from PID=181

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.