StorageGRID nodes losing connectivity with Node Down Alert
Applies to
- StorageGRID
- VMware Based Nodes
- StorageGRID Appliance
Issue
Some or all StorageGRID nodes report:
Unable to communicate with node
NODE_DOWN-CRITICAL
- Displays as
Unknown
Unexpected node reboot
Alert is marked as resolved after few minutes.
- From
Bycast.log
of affected nodes error reported:
Sep 29 14:31:29 <Nodename> ADE: |12536413 3008184556 RCON CDED 2023-09-29T14:31:29.759604| NOTICE 0603 580a14e60999bd82 RCON: peer 12415652 destroyed network result ENDT rcon type ndcd:DISCONNECTED reason dbrk:DISCONNECTED_BROKEN
Sep 29 14:31:29 <Nodename> ADE: |12536413 0518715227 RRTR ndcd 2023-09-29T14:31:29.759637| NOTICE 0700 RRTR: Connection to LDR, NID 12415652 was lost because "Neighbour connection was broken"