System down due to a dimm_uecc_error
Applies to
- FAS2820
- ONTAP 9.13.1
Issue
System down due to a dimm_uecc_error
EMS
<LR d="24Jan2025 10:50:28" n="node-02" t="xxxxxxxxxx" id="xxxxxxxxxx/xxxxxx" p="5" s="Ok" o="cf_hwassist" vf="" type="0" seq="328428" supp="9" >
<cf_hwassist_takeoverTrapRecv_1
partnerName="node-01"
trap_type="system_down"
trap_event="dimm_uecc_error"/>
</LR>
<LR d="24Jan2025 10:50:28" n="node-02" t="xxxxxxxxxx" id="xxxxxxxxxx/xxxxxx" p="5" s="Ok" o="cf_hwassist" vf="" type="0" seq="328429" >
<cf_hwassist_takeoverTrapRecv_1
partnerName="node-01"
trap_type="system_down"
trap_event="power_cycle_via_sp"/>
BMC log(system log)
Symptom 1
================ Log #2 start time Sun Jan 19 21:40:27 2025
BIOS Version: 19.2
Portions Copyright (C) 2014-2023 NetApp, Inc. All Rights Reserved.
Initializing System Memory ...
HALT! Memory Initialize Fatal Error.
Error Action: HALT System.
================ Log #6 start time Mon Jan 20 12:46:13 2025
BIOS Version: 19.2
Portions Copyright (C) 2014-2023 NetApp, Inc. All Rights Reserved.
Initializing System Memory ...
HALT! Memory Initialize Fatal Error.
Error Action: HALT System.
================ Log #8 start time Mon Jan 20 20:37:46 2025
BIOS Version: 19.2
Portions Copyright (C) 2014-2023 NetApp, Inc. All Rights Reserved.
Initializing System Memory ...
HALT! Memory Initialize Fatal Error.
Symptom 2
Local System ID: xxxxxxxxx
Press Ctrl-C for Maintenance menu to release disks.
Jan 24 10:50:27 [node-01:cf.fmns.skipped.disk:notice]: While releasing the reservations in "Waiting For Giveback" state Failover Monitor Node State(fmns) module skipped the disk 0b.00.1 that is owned by xxxxxxxxx and reserved by xxxxxxxxx.
Boot argument NfsRdmaOk not specified. Default (set) is in effect.
Boot argument NfsIpOk not specified. Default (set) is in effect.
FWU 2nd trigger point
FWU has no post firmware update action registered.
pnso provider init started.
Notice : pnso card not detected.
hwo: Node is using hardware provider : 1.
ECC error at DIMM-1: F47A2F80,ADDR 0x5a82d34c0,(Node(0), Memory controller(0), CH(0), DIMM(0), Rank(0), Bank Group(3), Bank(0x2), Row(0x9505), Col(0x350)) Uncorrectable Machine Check Error at CPU2. ICL_IMC0_C0 Error: STATUS<0xbe00004200a00090>(VALID,UC,EN,MISCV,ADDRV,PCC,CORR_ERR_STAT(0),CORR_ERR_CNT(0x1),OTHER_INFO(0x2),MSCOD(0xa0),MCACOD(0x90)),MISC<0x08000b04a829a886>(EXTRA_ERR_INFO(0x40005825414d4),ADDR_MODE(0x2),REC_ERR_LSB(0x6)),ADDR<0x00000005a82d34c0>(ADDRESS(0x5a82d34c0))Node(0), Memory controller(0), CH(0), DIMM(0), Rank(0), Bank Group(3), Bank(0x2), Row(0x9505), Col(0x350), Device(0), DQ-Burst(3,0),
Requesting SP to power cycle the filer to attempt to clear the Machine Check Event
…
ECC error at DIMM-1: F47A2F80,ADDR…
…
ECC error at DIMM-1: F47A2F80,ADDR…
…
ECC error at DIMM-1: F47A2F80,ADDR…
…
ECC error at DIMM-1: F47A2F80,ADDR…
…
Local System ID: xxxxxxxxx
Press Ctrl-C for Maintenance menu to release disks.
Jan 24 11:14:28 [node-01:cf.fmns.skipped.disk:notice]: While releasing the reservations in "Waiting For Giveback" state Failover Monitor Node State(fmns) module skipped the disk 0b.00.1 that is owned by xxxxxxxxx and reserved by xxxxxxxxx.
Boot argument NfsRdmaOk not specified. Default (set) is in effect.
Boot argument NfsIpOk not specified. Default (set) is in effect.
FWU 2nd trigger point
FWU has no post firmware update action registered.
pnso provider init started.
Notice : pnso card not detected.
hwo: Node is using hardware provider : 1.
ECC error at DIMM-1: F47A2F80,ADDR 0x5a82d36c0,(Node(0), Memory controller(0), CH(0), DIMM(0), Rank(0), Bank Group(3), Bank(0x2), Row(0x9505), Col(0x370)) Uncorrectable Machine Check Error at CPU0. ICL_IMC0_C0 Error: STATUS<0xfe00004200a00090>(VALID,OVER,UC,EN,MISCV,ADDRV,PCC,CORR_ERR_STAT(0),CORR_ERR_CNT(0x1),OTHER_INFO(0x2),MSCOD(0xa0),MCACOD(0x90)),MISC<0x08000b04a829b886>(EXTRA_ERR_INFO(0x40005825414dc),ADDR_MODE(0x2),REC_ERR_LSB(0x6)),ADDR<0x00000005a82d36c0>(ADDRESS(0x5a82d36c0))Node(0), Memory controller(0), CH(0), DIMM(0), Rank(0), Bank Group(3), Bank(0x2), Row(0x9505), Col(0x370), Device(0), DQ-Burst(3,4),
Requesting SP to power cycle the filer to attempt to clear the Machine Check Event