ANDU Paused on error. AFF-A400 Cluster ports hung during ONTAP upgrade

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 117

Visibility:: Public

Votes:: 0

Category:: aff-series

Specialty:: hw

Last Updated:

Applies to

AFF A400
Network Cluster ports (e3a and e3b)
Dual 100G Ethernet Controller IONIC in slot 3
Switchless Cluster Network

Issue

After node reboot during ONTAP upgrade process, cluster ports e3a and e3b do not come online on rebooted Node B
::> net interface show
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
Cluster
NODEA_CLUSINTC1
up/up 169.254.76.105/16 NODEA e3a true
NODEA_CLUSINTC2
up/up 169.254.174.137/16 NODEA e3b true
NODEB_CLUSINTC1
up/- 169.254.227.210/16 NODEB e3a true

NODEB_CLUSINTC2
up/- 169.254.13.40/16 NODEB e3b true
Updated Node B remains in partial giveback
::*> storage failover show
Takeover
Node Partner Possible State Description
-------------- -------------- -------- -------------------------------------
NODEA NODEB false Connected to NODEB, Partial
giveback, Takeover is not possible:
The version of software running on
each node of the SFO pair is
incompatible, NVRAM log not
synchronized
NODEA NODEB - Waiting for cluster applications to
come online on the local node
Offline applications: mgmt, vldb,
vifmgr, bcomd, crs, scsi blade, clam.
2 entries were displayed.
Surviving Node A, not updated, shows one cluster port offline and the other cluster port online
Loopback test on Node B indicates network card is failed
- Operational link is not observed during the loopback at physical port or in ONTAP CLI outputs
Loopback test on Node A does not change the ports behaviour
Power cycling of the updated (down) Node B does not bring the ports back online
Attempting to bounce ports in Node A does not solve the issue

One of the ports in the surviving Node A indicates it is Online, despite being physically disconnected (port hung)

Port shows "Online" in ONTAP CLI through outputs for commands ::> network port show and ::> node run -node NodeA -command sysconfig -a

######cluster ports up/running
             slot 3: 100G Ethernet Controller IONIC
                e3a MAC Address:    00:ae:cd:09:b8:20 (auto-100g_cr4-fd-up)
                    QSFP Vendor:         Amphenol
                    QSFP Part Number:    112-00595
                    QSFP Serial Number:  APF20339236111
                e3b MAC Address:    00:ae:cd:09:b8:21 (auto-100g_cr4-fd-up)
                    QSFP Vendor:         Amphenol
                    QSFP Part Number:    112-00595
                    QSFP Serial Number:  APF20339236130
                Device Type:        ionic
                Firmware Version:   1.0.1-E-31
                Serial Number:      FPN20370049

###### node2 output looks like ports e3a/e3b down
    slot 3: Dual 100G Ethernet Controller IONIC
                e3a MAC Address:    00:ae:cd:09:ba:00 (auto-unknown-down)
                    QSFP Vendor:         Amphenol
                    QSFP Part Number:    112-00595
                    QSFP Serial Number:  APF20339236111
                e3b MAC Address:    00:ae:cd:09:ba:01 (auto-unknown-down)
                    QSFP Vendor:         Amphenol
                    QSFP Part Number:    112-00595
                    QSFP Serial Number:  APF20339236130
                Device Type:        ionic
                Firmware Version:   1.4.0-E-114
                Serial Number:      FPN2037005D