Skip to main content
NetApp Knowledge Base

StorageGRID appliance SG5700 reboots unexpectedly due to watchdog timeout

Views:
465
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

  • NetApp StorageGRID 11.5 and above
  • StorageGRID Appliances SG5700

Issue

  • Alert email of unexpected reboot for one or more nodes:
Unexpected node reboot (1 alert) 
A node rebooted unexpectedly within the last 24 hours.
Recommended actions
1. Monitor this alert. The alert will be cleared after 24 hours. However, if the node reboots unexpectedly again, this alert will be triggered again.
2. If you cannot resolve the alert, there might be a hardware failure. Contact technical support.
________________________________________
dc1-sn1 
Node    dc1-sn1
Site    DC1
Severity    Major
Time triggered    WKD MMM DD hh:mm:ss UTC YYYY
Job    miscd
  • High CPU utilization reported on the node in question leading up to the reboot. The CPU usage can be viewed under the Nodes view in StorageGRID Grid Manager, and under the (Support > Metrics) Node or Node (Internal Use) views.
  • The node is rebooted due to watchdog timeout:
    • Collecting log files and system data on affected nodes
    • Extract the .tar.gz log file
    • Locate the crash dmesg file: base-os-logs\run\mount-tmp\pge-actv-root\var\log\storagegrid_crash_dmesg.YYYYMMDDhhmmss.log.gz
    • Extract and open the crash dmesg file, verify the node is rebooted due to watchdog timeout:

[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr fired, interruptCount = 1
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging memory usage
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging CPU backtrace
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging blocked tasks
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging all ftrace buffers
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr serviced watchdog

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.