Volumes getting high latency due to incomplete decommission of StorageGRID node
Applies to
- StorageGRID 11.7.0
- Decommission of VM-based node
Issue
- Host volumes having high latency due to incomplete decommission of storageGRID Node.
- We could see Decommission completed on Grid Topology > Site > Primary Admin node > CMN > Tasks but From Maintenance > Decommission page doesnt reset.
- Decommissionned Node logs shows it got failed, and although it should re-try the decommission on this node but instead got success message (
Child process exited with RC=0
) due to a known issue./log/cassandra/debug.log
2024-08-26 20:09:32,835 StorageService.java (line 4253) DECOMMISSIONING
ERROR [RMI TCP Connection(343039)-127.0.0.1] 2024-08-26 21:27:30,048 StorageService.java (line 4324) Error while decommissioning node
java.lang.RuntimeException: Failed to transfer all hints to 2ee03617-a26c-48ee-8f3e-7b83ec5b039c
....
Aug 26 21:28:30 sg-hydprod-t2-node01 ADE: |21938310 2079159933 DDSM GASK 2024-08-26T21:28:30.367748| NOTICE 1541 DDSM: Child process 3482633 exited with RC=0