Troubleshooting LACP port channel/ interface groups

What is a Port Channel Group?

Port Channel Group is a set of multiple physical Ethernet ports aggregated together for the purposes of increased aggregate throughput and/or for improved network resiliency. It is also known as an etherchannel, trunk, port bundle, or LACP. The Institute of Electrical and Electronics Engineers (IEEE) defined the standard for port channel groups with designation 802.3ad and 802.3ax, while port channel group refers commonly to a switch side configuration, NetApp uses the term 'interface groups / ifgrp' or the legacy name of Virtual Interface (VIF). Interface groups on the NetApp side come in three flavors.

• SystemIDs: Each member in the lag will send a systemID in its LACPDUs. All the member ports for the filer should send the same systemID to indicate that only one logical device is connected to the switch. Similarly, each of the switch's member ports should send the same systemID. However, it is expected that the systemID will differ between the switch and filer.

  • If the member ports don't send the same systemID, then one side is trying to aggregate two different logical devices which is not supported with LACP. For example, a Cisco Nexus cross-switch link-aggregation group may send two different systemIDs if a virtual port-channel (VPC) isn't configured on the switch. Configuring the link-aggregation group with a VPC tells each switch to send the same SystemID. 

    • NOTE:  See example of IFCONFIG -A below showing mis-matched systemIDs

• Single Mode: No switch side configuration, active/ passive only, and not a port channel group
• Multimode: A 'static' port channel group. The storage controller and switch are hard-coded with a set number of ports that will always be members of a port channel group. It is less optimal than Multimode_LACP because the storage controller and switch cannot prevent an individual port from participating in the port channel group unless the port goes offline (example: no cable plugged in).
  
  If the device configured with this type of port channel group is cabled incorrectly, the 'static' multimode port channel group would also be unaware of the mis-cabling, and would use all ports for transmit. You should then see CAM table flapping ('MAC flapping') reported by the switch. Also, the network connectivity, if established, would be very inconsistent.
  
  Note: A switch administrator might refer the static port channel group as LACP, but a 'static' multimode ifgrp or port channel group does not use the LACP protocol, and the two terms cannot be used interchangeably.
  
  Multimode_LACP: A port channel group mode that allows two network devices (for example, a switch and NetApp storage controller) to communicate and compare port state and parameters. Because LACP is a communication between two ports (for example, a NIC port on a server or client and the switch port it is plugged into), LACP is able to confirm that communications between the two devices are successful. It allows either participant to make a decision about whether or not each physical port in the port-channel group should be used. This is superior to 'static' multimode port channel groups only in that it can detect some conditions that are not associated with an unplugged or totally failed port. ONTAP requires that each port channel can only have one SystemID and this cannot be spanned across multiple switches.
  
  There are two 'modes' of operation in an LACP port:
• Active LACP ports always participate in LACP negotiations. If the interface in the neighboring device (the other end of the cable) is not configured for LACP use (active or passive), then this port will be disabled.
• Passive LACP ports are allowed to participate in LACP negotiations, if the device at the other end of the cable initiates by sending an LACP control packet.
• All NetApp storage controllers always operate in 'Active LACP' mode. This is non-configurable.
  LACP will be operating in one of two 'timings' at any given time:
• 'Slow timers' exchange LACP control packets once every 30 seconds.
• 'Fast timers' exchange LACP control packets once every 1 second.
• It is expected that most devices will use 'Fast timers' when a port is initially brought 'up'. It is also expected that LACP will move to 'Slow timers' soon after LACP has properly negotiated, and the ports come up for use.
  LACP can disable a member of the port channel under several circumstances.
• If the network device stops receiving updates from the other device, the port will eventually be disabled. This takes up to 90 seconds (it can be less).
• If the network device receives updates from the neighboring device, but that update includes incorrect information, the network device can disable the port. This will be expected to detect mis-cabled systems, and can occur in far less than 90 seconds.
• If the network device receives an update that indicates that the neighboring port is not useable, the port should be disabled immediately upon receiving that LACP control packet.
• In some cases, a switch set to 'Passive' will not negotiate correctly with the storage controller, even though the storage controller is set to 'Active' by default.
• In some cases, a switch might not move to 'Slow timers' when the storage controller moves. This can prevent LACP from bringing the ports into service.
  
  Note: LACP will only be able to confirm that two neighboring devices (example: NIC and Switch) are communicating directly with each other. Any other failures, such as routing failures or switch outages not affecting this port, will not be detected by LACP.

Who creates Port Channel Groups?

Storage architects and network administrators have to work together to ensure their port channel groups are configured correctly. If this collective validation is not done, the network infrastructure might be at risk.

Troubleshooting Port Channel Groups/Ifgrps