System does not start after reboot with Unable to recover the local database of Data Replication Module
- Views:
- 15,812
- Visibility:
- Public
- Votes:
- 7
- Category:
- fas-systems
- Specialty:
- hw
- Last Updated:
- 4/2/2025, 2:14:49 PM
Applies to
- All AFF and FAS platforms
- ONTAP 9
- Power outage, controller relocation, or during upgrade or other reboot
Issue
- Console logs for node show:
******************************************************
* This is a serial console session. Output from this *
* session is mirrored on the SP console session. *
******************************************************
***********************
** SYSTEM MESSAGES **
***********************
Internal error: Cannot open corrupt replicated database. Automatic recovery
attempt has failed or is disabled. Check the event logs for details. This node
is not fully operational. Contact support personnel for the root volume recovery
procedures.
- EMS messages for node:
Mar 28 17:03:46 [NODE_B:rdb.recovery.failed:EMERGENCY]: Error: Unable to find a master. Unable to recover the local database of Data Replication Module: Management.
Mar 28 17:03:46 [NODE_B:spm.mgwd.process.exit:EMERGENCY]: Management Gateway (mgwd) subsystem with ID 1944 exited as a result of signal normal exit (0). The subsystem will attempt to restart.
- Node boots and it is possible to log in but the cluster commands are not showing the correct outputs:
::> cluster show
Error: "show" is not a recognized command
::> set advanced
::*> cluster ring show
Error: "show" is not a recognized command
ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED
error message is displayed during the boot/login phase.- Some
bootarg
values are stated forcorrupt
and/orrecovery
in the LOADER. Example:
LOADER-B> printenv bootarg.rdb_corrupt
Variable Name Value
-------------------- --------------------------------------------------
bootarg.rdb_corrupt 0500000000
LOADER-B> printenv bootarg.init.boot_recovery
Variable Name Value
-------------------- --------------------------------------------------
bootarg.init.boot_recovery 80
Note: If bootarg is not set, it is displayed as undefined.