VIFMGR offline migrating cluster switch CN1610 to BES-53248
Applies to
- Cluster Network Switch (CNS) hardware upgrade
- CN1610
- BES-53248
Issue
- Several LIFs in operational status down.
- Downtime in several services for the SVMs affected.
- Cluter ping fails due to disabled ISLs between switches while still connecting the nodes.
- VIFMGR offline, some interfaces migrated or operational status down. Example:
Wed Jun 23 2021 09:27:18 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] 100% packet loss when pinging from node1_lif1 ( 169.254.137.239 ) on node1 -> node3_lif2 ( 169.254.16.239 ) on node3
Wed Jun 23 2021 09:27:19 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] l2ping failed e0a on node1 -> e0b on node3 result='remote interface is down or unreachable. If a VLAN was specified, it may not be configured on the switch.'
Wed Jun 23 2021 09:27:42 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] 100% packet loss when pinging from node1_lif1 ( 169.254.137.239 ) on node1 -> node4_lif2 ( 169.254.128.136 ) on node4
Wed Jun 23 2021 09:27:44 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] l2ping failed e0a on node1 -> e0b on node4 result='remote interface is down or unreachable. If a VLAN was specified, it may not be configured on the switch.'
Wed Jun 23 2021 09:28:07 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] 100% packet loss when pinging from node1_lif1 ( 169.254.137.239 ) on node1 -> node2_lif2 ( 169.254.63.209 ) on node2
Wed Jun 23 2021 09:28:08 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] l2ping failed e0a on node1 -> e0c on node2 result='remote interface is down or unreachable. If a VLAN was specified, it may not be configured on the switch.'
Wed Jun 23 2021 09:28:32 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] 100% packet loss when pinging from node1_lif1 ( 169.254.137.239 ) on node1 -> node2_lif2 ( 169.254.136.171 ) on node2
Wed Jun 23 2021 09:28:33 +02:00 [kern_vifmgr:info:7192] [0x80bae2900] [FailoverMgr::cluster_check] l2ping failed e0a on node1 -> e0c on node2 result='remote interface is down or unreachable. If a VLAN was specified, it may not be configured on the switch.'
...
Wed Jun 23 2021 09:35:03 +02:00 [kern_vifmgr:info:6719] [0x80b970b00] [Net::RdbLifHandle::avoidDownPorts] LIF lif:rdb:4294967293:node2_lif2 (947) is assigned to a down port (node2:e0c). Attempting to reassign.
...
Wed Jun 23 2021 09:35:14 +02:00 [kern_vifmgr:info:6719] A [src/rdb/TM.cc 1760 (0x80bf51600)]: _triggerOnlineStatusCallback: FAILOVER rdb: Local unit VifMgr offline
Wed Jun 23 2021 09:35:14 +02:00 [kern_vifmgr:info:6719] Notice: online_status_callback: RDB unit is offline
...
Wed Jun 23 2021 09:35:14 +02:00 [kern_vifmgr:info:6719] [0x80b970b00] [EventMgr::workerThreadMain] received port table update while OFFLINE
...
Wed Jun 23 2021 09:35:18 +02:00 [kern_vifmgr:info:6719] A [src/rdb/TM.cc 1787 (0x810d52700)]: _triggerOnlineStatusCallback: FAILOVER rdb: Local unit VifMgr online
Wed Jun 23 2021 09:35:18 +02:00 [kern_vifmgr:info:6719] A [src/rdb/cluster_events.cc 88 (0x810d52700)]: Report: Cluster event: node-event, epoch 19, site 1007 [local node online (secondary)].
Wed Jun 23 2021 09:35:18 +02:00 [kern_vifmgr:info:6719] Notice: online_status_callback: RDB unit is online as Secondary