Skip to main content
NetApp Knowledge Base

StorageGRID appliance kernel panic after Correctable ECC Error Logging Limit Reached

Views:
127
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

StorageGRID appliance SG6000

Issue

  • StorageGRID detects Unable to communicate with node alert due to storage node reboot
  • storagegrid_crash_dmesg.<TIMESTAMP>.log shows Kernel panic after many I/O errors

[61824251.190163] sd 17:0:0:240: rejecting I/O to offline device
[61824251.193978] sd 17:0:0:238: rejecting I/O to offline device
[61824251.193982] sd 17:0:0:246: rejecting I/O to offline device
[61824251.193984] sd 17:0:0:244: rejecting I/O to offline device
[61824251.193985] sd 17:0:0:242: rejecting I/O to offline device
[61824251.193994] device-mapper: multipath: Failing path 68:0.
[61824251.193996] device-mapper: multipath: Failing path 67:224.
[61824251.194005] device-mapper: multipath: Failing path 67:192.
[61824251.194039] sd 6:0:0:242: rejecting I/O to offline device
[61824251.194043] device-mapper: multipath: Failing path 8:96.
[61824251.194056] sd 15:0:0:242: rejecting I/O to offline device
[61824251.194058] device-mapper: multipath: Failing path 65:128.
[61824251.194078] XFS (dm-11): metadata I/O error in "xlog_iodone" at daddr 0x27c26a78 len 64 error 5
[61824251.194080] XFS (dm-11): xfs_do_force_shutdown(0x2) called from line 1271 of file fs/xfs/xfs_log.c.  Return address = 00000000557f8fa2
[61824251.194140] XFS (dm-11): Log I/O Error Detected.  Shutting down filesystem
[61824251.194140] XFS (dm-11): Please umount the filesystem and rectify the problem(s)
[61824251.194151] sd 6:0:0:244: rejecting I/O to offline device
[61824251.194153] device-mapper: multipath: Failing path 8:128.
[61824251.194162] sd 15:0:0:244: rejecting I/O to offline device
[61824251.194164] device-mapper: multipath: Failing path 65:160.
[61824251.194187] XFS (dm-13): metadata I/O error in "xlog_iodone" at daddr 0x27c39778 len 64 error 5
[61824251.194191] XFS (dm-13): xfs_do_force_shutdown(0x2) called from line 1271 of file fs/xfs/xfs_log.c.  Return address = 00000000557f8fa2
[61824251.194193] sd 6:0:0:246: rejecting I/O to offline device
[61824251.194195] device-mapper: multipath: Failing path 8:160.
[61824251.194202] sd 15:0:0:246: rejecting I/O to offline device
[61824251.194203] device-mapper: multipath: Failing path 65:192.
[61824251.194216] XFS (dm-15): metadata I/O error in "xlog_iodone" at daddr 0x27c06fa8 len 64 error 5
[61824251.194218] XFS (dm-15): xfs_do_force_shutdown(0x2) called from line 1271 of file fs/xfs/xfs_log.c.  Return address = 00000000557f8fa2
[61824251.194242] XFS (dm-15): Log I/O Error Detected.  Shutting down filesystem
[61824251.194243] XFS (dm-15): Please umount the filesystem and rectify the problem(s)
[61824251.194245] XFS (dm-13): Log I/O Error Detected.  Shutting down filesystem
[61824251.194246] XFS (dm-13): Please umount the filesystem and rectify the problem(s)
[61824251.199608] sd 15:0:0:251: rejecting I/O to offline device
[61824251.205849] device-mapper: multipath: Failing path 67:128.
[61824251.212037] sd 15:0:0:251: rejecting I/O to offline device
[61824251.222021] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

  • SEL_TEXT.txt downloaded by BMC Logs & Reports IPMI Event Log > Download Event Logs by selecting Text File Type shows Correctable ECC error after Logging Limit Reached

Event ID        Time Stamp                    Severity    Sensor Name            Sensor Type                Description
    46        <TIMESTAMP>    [Information]    [Memory Error]    [Memory]        Correctable ECC(CPU0_E0) - Asserted
    45        <TIMESTAMP>    [Information]    [Memory Error]    [Memory]        Correctable ECC(CPU0_E0) - Asserted
    44        <TIMESTAMP>        [Information]    [Memory Error Dis]    [Event Logging Disabled]    Correctable Memory Error Logging Disabled - Asserted
    43        <TIMESTAMP>    [Critical]        [Memory Error]    [Memory]        Correctable ECC Error Logging Limit Reached(CPU0_E0) - Asserted
​​​​​

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.