Node panic after migration from switched to switchless cluster
Issue
- New nodes added to cluster and data migrated via volume move.
- After migration completed, old nodes removed from cluster and cluster transitioned from switched to switchless.
- During this process, cables changed from 40G copper to 100G copper on cluster ports e0a and e1a.
- After this swap, large number of CRC errors and long frames reported on both cluster ports on one node.
- NodeIfInErrorsWarnAlert reported and system health subsystem show reports degraded cluster.
- Node reporting CRC errors panics several times:
PANIC: Unknown key type 0 in SK process wafl_exempt18 on release 9.11.1P14 (C) on Sat Mar 2 00:27:34 IST 2024
PANIC: protection fault on VA 0 code 0 cs:rip 0x20:0xffffffff8a950af0 in process NwkThd_00 on release 9.11.1P14 (C) on Sat Mar 2 02:44:58 IST 2024