Correctable memory errors reporting in ONTAP versions with dynamic thresholds
Applies to
ONTAP versions:
- 9.5 and later major releases
- 9.4P6 and later P releases
- 9.3P11 and later P releases
- 9.1P18 and later P releases
Platforms:
- AFF systems
- ASA systems
- FAS systems
- CriticalCECCCountMemErrAlert
Note: For all other ONTAP versions see How to troubleshoot correctable memory errors on FAS and AFF systems
Issue
- Health Alert AutoSupport message:
HA Group Notification (Health Monitor process nphm: CriticalCECCCountMemErrAlert[DIMM-16]) ALERT
- EMS reports:
[mgwd: callhome.hm.alert.critical:alert]: Call home for Health Monitor process nphm: CriticalCECCCountMemErrAlert[DIMM-16].
- Management Log
[nphmd: hm.alert.raised:alert]: Alert Id = CriticalCECCCountMemErrAlert , Alerting Resource = DIMM-16 raised by monitor controller
- Health Monitor reports
::> system health alert show
Node: Node1
Alert ID: CriticalCECCCountMemErrAlert
Resource: DIMM-1
Severity: Critical
Indication Time: Wed Jan 10 16:45:18 2024
Suppress: false
Acknowledge: false
Probable Cause: The DIMM has degraded, leading to memory errors.
Possible Effect: Memory issues can lead to a catastrophic system panic,
which can lead to data downtime on the node.
Corrective Actions:
1. Contact technical support to obtain a new DIMM of the same specification.
2. If possible, perform a takeover of this node and bring the node down for maintenance.
3. Refer to the DIMM replacement guide for your given hardware platform to replace the DIMM.
4. Bring the storage system online.
Alertdditional Information
Slot Name: DIMM-16
CPU Socket: 0
Channel: 0
Slot Number on a Channel: 0
Correctable ECC Error Count: 329793
Uncorrectable ECC Error Count: 0
Correctable ECC Error Limit: 0
CECC Alert Method: bucket
Correctable ECC Replace DIMM: true
Failure Reason: none
Correctable ECC Error Minimum Count: 325104
Correctable ECC Error Bucket Count: 177243
Correctable ECC Replace DIMM Count: 3
DIMM Serial Number: *****
Node Serial Number: *****
Node Model: FAS****
Alerting Resource Name DIMM-16
- Memory dimm show reports the dimm status as degraded
Cluster::> memory dimm show
DIMM UECC CECC Alert CPU Slot Failure
Node Name Count Count Method Socket Channel Number Status Reason
------------- ------- ----- ----- ------ ------ ------- ------ ------- --------
Cluster-01
DIMM-1 0 0 bucket 1 0 0 ok none
DIMM-2 0 0 bucket 1 0 1 ok none
DIMM-3 0 0 bucket 1 1 0 ok none
DIMM-4 0 0 bucket 1 1 1 ok none
DIMM-5 0 0 bucket 1 3 1 ok none
DIMM-6 0 0 bucket 1 3 0 ok none
DIMM-7 0 0 bucket 1 2 1 ok none
DIMM-8 0 0 bucket 1 2 0 ok none
DIMM-9 0 0 bucket 0 2 0 ok none
DIMM-10 0 0 bucket 0 2 1 ok none
DIMM-11 0 0 bucket 0 3 0 ok none
DIMM-12 0 0 bucket 0 3 1 ok none
DIMM-13 0 0 bucket 0 1 1 ok none
DIMM-14 0 0 bucket 0 1 0 ok none
DIMM-15 0 0 bucket 0 0 1 ok none
DIMM-16 0 0 bucket 0 0 0 degraded none
- Performance issues may be seen when errors are seen in sktrace in AutoSupport logs
CECC_LOG_ENTRY: Number of correctable ECC since boot 355314194: Information about Correctable ECC: ECC error at DIMM 8: CE-12-1234-12345678,ADDR 0xaada0eec0, (Node(0), Memory controller(1), CH(2), DIMM(0), Rank(0), Bank Group(3), Bank(0x2), Row(0x2174), Col(0x1d8))