- All versions of ONTAP
- Any FAS and AFF platform other than the FAS25XX, FAS22XX, FAS/V32XX, and FAS/V62XX.
AnswerFor all ONTAP FAS and AFF platforms other than the FAS25XX, FAS22XX, FAS/V32XX, and FAS/V62XX, this article supersedes Knowledgebase article How to troubleshoot correctable memory errors.
Why is NetApp changing correctable memory error monitoring in ONTAP?
NetApp storage systems utilize error-correcting code (ECC) memory modules (DIMMs) for both main system memory and NVRAM/NVMEM subsystems. When possible, memory errors are corrected in-flight by the memory subsystem hardware with little to no impact on system performance. Until recently, ONTAP running on AFF/FAS storage systems employed a longstanding policy to alert the system administrator about “excessive” CECC memory errors based on a threshold of 500 errors since the last reboot of the system.
After recent extensive analysis of correctable ECC (CECC) memory errors by NetApp and its hardware component vendors, it was determined that CECC memory errors are typically not a good predictor of a system disruption due to uncorrectable ECC (UECC) memory errors – especially with the latest generations of memory controllers and dynamic random-access memory (DRAM). Additionally, the CPU cycles used to monitor, log and correct large numbers of memory errors have negligible impact to system performance.
As a result, NetApp has changed the monitoring algorithm for CECC memory errors used by ONTAP on many currently-supported AFF/FAS systems to a dynamic monitoring algorithm, with much higher thresholds configured to trigger the “CriticalCECCCountMemErrAlert” controller Health Monitor alert and corresponding "Health Monitor" AutoSupport message.
Alerts triggered using the older policy can be considered false positives and should not be taken as an indication for memory replacement as it will result in unnecessary hardware maintenance with no tangible benefits.
NVRAM DIMMs do have correctable ECC replacement guidelines that are not ONTAP version specific. For more information refer to the replacement guidelines matrix.
With the dynamic monitoring algorithm in place, how do I determine when a DIMM needs to be replaced due to excessive CECC memory errors?
Refer to the table below for memory replacement guidelines.
|ECC Type||Category||Replacement Criteria|
ONTAP versions prior to:
HA Group Notification from cluster-01 (Health Monitor process nphm: CriticalCECCCountMemErrAlert[DIMM-14]) ALERT.
NVRAM DIMM, all ONTAP versions
NVRAM DIMMs are a FRU (except for NVRAM10P). Replace the NVRAM DIMM (or NVRAM10P card) when the CECC count is greater than 2 per week or greater than 5 per month.
The NV CECC error counter name varies by NVRAM type:
You can also read the NV CECC error counter (last-sensor-value) in the PLATFORM-SENSORS.XML AutoSupport
|Uncorrectable (UECC) Panic||All platforms, ONTAP versions||DIMM associated with the panic should be replaced.|
- On versions of ONTAP that use the dynamic algorithm, CECC memory errors continue to be periodically logged in ONTAP event logs. However, they are no longer relevant in determining the need for DIMM replacement.
- Correctable ECC errors are not an indicator that an uncorrectable ECC error will occur. Should an uncorrectable memory error occur, it will cause a system disruption (panic). If a system disruption occurs, the panic message will call out the DIMM or DIMMs where the uncorrectable error occurred. Those DIMMs should then be replaced.
- Recent BIOS/LOADER releases for current shipping ONTAP platforms contain memory management enhancements. These updates improve resiliency to uncorrectable ECC errors as well as reduce scenarios where DIMMs can be mapped out during boot such as Bugs 1195242, 1195243, or 1195423. If your BIOS version is not the latest available for your AFF or FAS system, NetApp recommends updating the BIOS to the latest version. Find the latest BIOS/LOADER version for your systems here: https://mysupport.netapp.com/NOW/cgi-bin/fw.