CONTAP-560034: Intel NICs communication failure with Cisco Nexus 5K/7K switch after ONTAP 9.18.1 upgrade

Issue

• After upgrading to ONTAP 9.18.1 or later, affected XL710/X710 NIC ports may lose connectivity through Cisco Nexus 5K/7K switches.

|Affected Environment :| * Intel XL710 NICs: X91440A

• Intel X710 NICs: X1147A
• Cisco Nexus 5K series (e.g., Nexus 5596) running any NX-OS version
• Cisco Nexus 7K series (e.g., Nexus 7018) running any NX-OS version
• Single-port or IFGRP/LACP network configurations|

 

• Traffic fails to pass through the switch despite physical links remaining up.
• If the affected LIFs use LACP mode,  all zeroes for the lag id in ifconfig -v

Example of ifconfig -v 
a0c: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 02:a0:98:a3:a9:1c
laggproto lacp lagghash l3
lagg options:
flags=1<USE_FLOWID>
flowid_shift: 16
lagg statistics:
active ports: 0
flapping: 1lag id: [(0000,00-00-00-00-00-00,0000,0000,0000), (0000,00-00-00-00-00-00,0000,0000,0000)]


 

• A typical symptom is that NIC transmit queues continuously enqueue packets without dequeuing them, as shown below: 

::> systemshell -node <node name> -command sysctl dev.ixl | grep txq_ | grep queued | grep txq00
i.e.
dev.ixl.1b.iflib.txq00.txq_enqueued: 1920
dev.ixl.1b.iflib.txq00.txq_dequeued: 0
dev.ixl.1a.iflib.txq00.txq_enqueued: 1920
dev.ixl.1a.iflib.txq00.txq_dequeued: 0

• This indicates packets are being queued for transmission but are not being successfully transmitted through the switch
• Nexus-side logs:  

LACP_ST_SUSPENDED_FOR_MISCONFIGURATION
LACP_EV_RECEIVE_PARTNER_PDU_TIMED_OUT