How to troubleshoot correctable memory errors on FAS and AFF systems
Applies to
- ONTAP versions
- 9.5 and later releases
- 9.4P6 and later 9.4 releases
- 9.3P11 and later 9.3 releases
- 9.1P18 and later 9.1 releases
- Any FAS and AFF platform other than the FAS25XX, FAS22XX, FAS/V32XX, and FAS/V62XX.
Answer
For all ONTAP FAS and AFF platforms other than the FAS25XX, FAS22XX, FAS/V32XX, and FAS/V62XX, this article supersedes Knowledgebase article How to troubleshoot correctable memory errors.
- Why is NetApp changing correctable memory error monitoring in ONTAP?
- NetApp storage systems utilize error-correcting code (ECC) memory modules (DIMMs) for both main system memory and NVRAM/NVMEM subsystems. When possible, memory errors are corrected in-flight by the memory subsystem hardware with little to no impact on system performance.
- Until recently, ONTAP running on AFF/FAS storage systems employed a longstanding policy to alert the system administrator about “excessive” CECC memory errors based on a threshold of 500 errors since the last reboot of the system.
- After recent extensive analysis of correctable ECC (CECC) memory errors by NetApp and its hardware component vendors, it was determined that CECC memory errors are typically not a good predictor of a system disruption due to uncorrectable ECC (UECC) memory errors – especially with the latest generations of memory controllers and dynamic random-access memory (DRAM).
- Additionally, the CPU cycles used to monitor, log and correct large numbers of memory errors have negligible impact to system performance.
- As a result, NetApp has changed the monitoring algorithm for CECC memory errors used by ONTAP on many currently-supported AFF/FAS systems to a dynamic monitoring algorithm, with much higher thresholds configured to trigger the “CriticalCECCCountMemErrAlert” controller Health Monitor alert and corresponding "Health Monitor" AutoSupport message.
- Alerts triggered using the older policy can be considered false positives and should not be taken as an indication for memory replacement as it will result in unnecessary hardware maintenance with no tangible benefits.
- NVRAM DIMMs do have correctable ECC replacement guidelines that are not ONTAP version-specific. For more information, refer to the replacement guidelines matrix.
- With the dynamic monitoring algorithm in place, how do I determine when a DIMM or NVDIMM needs to be replaced due to excessive correctable or uncorrectable memory errors?
Refer to the table below for memory replacement guidelines:
ECC Type | Category | Replacement Criteria | |
Correctable (CECC) (Dynamic algorithm) |
DIMM, NVDIMM ONTAP versions:
|
Example: HA Group Notification from cluster-01 (Health Monitor process nphm: CriticalCECCCountMemErrAlert[DIMM-14]) ALERT.
|
|
Correctable (CECC) |
DIMM ONTAP versions prior to:
|
|
|
Correctable (CECC) |
NVRAM DIMM, all ONTAP versions
|
Raw <asup:ROW col_time_us="3423606694499"> |
|
Uncorrectable (UECC) Panic or Uncorrectable Machine Check Error (UMCE) Panic |
All platforms, ONTAP versions |
Check the system console log from the SP or BMC. Examine the console log for panic message details and Post Package Repair (PPR) operation results. If PPR information is not available, replace the DIMM associated with the panic. If PPR information results are available:
See: BIOS updates for memory reliability and the PPR feature |
Check Active IQ to see if CECC memory impact your systems.
Notes:
- On versions of ONTAP that use the dynamic algorithm, CECC memory errors continue to be periodically logged in ONTAP event logs. However, they are no longer relevant in determining the need for DIMM replacement.
- Correctable ECC errors are not an indicator that an uncorrectable ECC error will occur. Should an uncorrectable memory error occur, it will cause a system disruption (panic). If a system disruption occurs, the panic message will call out the DIMM or DIMMs where the uncorrectable error occurred. Those DIMMs might need to be replaced (see table, above).
- Recent BIOS/LOADER releases for current shipping ONTAP platforms contain memory management enhancements. These updates improve resiliency to uncorrectable ECC errors as well as reduce scenarios where DIMMs can be mapped out during boot such as Bugs 1195242, 1195243, or 1195423. If your BIOS version is not the latest available for your AFF or FAS system, NetApp recommends updating the BIOS to the latest version. Find the latest BIOS/LOADER version for your systems on the System Firmware & Diagnostics Download page.
- JEDEC-standard NVDIMM modules are used in the following platforms:
- AFF A800, AFF A400, AFF A320
- FAS8700, FAS8300