VLDB offline on the node due to huge WAL file size on the partner node
Applies to
Issue
- Node goes down due to a panic and boots up.
- After the boot, the VLDB is offline on the node:
::> set advanced
::*> cluster ring show
Node UnitName Epoch DB Epoch DB Trnxs Master Online
--------- -------- -------- -------- -------- --------- ---------
Node-01 mgmt 35 35 130 Node-01 master
Node-01 vldb 21 21 932755 Node-01 master
Node-01 vifmgr 37 37 79 Node-01 master
Node-01 bcomd 34 34 3 Node-01 master
Node-01 crs 35 35 1 Node-01 master
Node-02 mgmt 35 35 130 Node-01 secondary
Node-02 vldb 0 21 932644 - offline
Node-02 vifmgr 37 37 79 Node-01 secondary
Node-02 bcomd 34 34 3 Node-01 secondary
Node-02 crs 35 35 1 Node-01 secondary
10 entries were displayed.
- Partner node VLDB Data Grows to Gigabyte Level.
::> set diag
::*> systemshell * "ls -lh /mroot/etc/cluster_config/rdb/*/_sql/rdb.db-wal"
Node: Node-01
-rw-r--r-- 1 root wheel 676K Nov 10 13:36 /mroot/etc/cluster_config/rdb/Bcom/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 255K Nov 10 13:36 /mroot/etc/cluster_config/rdb/Crs/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 2.1M Nov 10 13:38 /mroot/etc/cluster_config/rdb/Management/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 19G Nov 10 13:39 /mroot/etc/cluster_config/rdb/VLDB/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 2.1M Nov 10 13:38 /mroot/etc/cluster_config/rdb/VifMgr/_sql/rdb.db-wal
Node: Node-02
-rw-r--r-- 1 root wheel 10K Nov 10 12:48 /mroot/etc/cluster_config/rdb/Bcom/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 4.1K Nov 10 12:48 /mroot/etc/cluster_config/rdb/Crs/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 427K Nov 10 13:15 /mroot/etc/cluster_config/rdb/Management/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 6.1K Nov 10 13:11 /mroot/etc/cluster_config/rdb/VLDB/_sql/rdb.db-wal
-rw-r--r-- 1 root wheel 641K Nov 10 13:15 /mroot/etc/cluster_config/rdb/VifMgr/_sql/rdb.db-wal