What are the flow control best practices for Ethernet?
Applies to
Data ONTAP
Answer
Best Practices:
Follow flow control best practices when experiencing excessive pause frames, xon / xoff frames, or packets being re-queued, as seen in ifstat -av.
- interface e1a (0 hours, 0 minutes, 40 seconds) -- RECEIVE Frames/second: 1568 | Bytes/second: 27607k | Errors/minute: 0 Discards/minute: 0 | Total frames: 248k | Total bytes: 4222m Total errors: 0 | Total discards: 0 | Multi/broadcast: 2 No buffers: 0 | Non-primary u/c: 0 | LRO segments: 172k LRO bytes: 3929m | L2 terminate: 0 | Tag drop: 0 Vlan tag drop: 0 | Vlan untag drop: 0 | Vlan forwards: 0 Vlan broadcasts: 0 | Vlan unicasts: 0 | CRC errors: 0 Runt frames: 0 | Fragment: 0 | Long frames: 0 Jabber: 0 | Bus overruns: 0 | Queue drop: 0 Xon: 25 | Xoff: 25 | Jumbo: 68353 No buf hi: 0 TRANSMIT Frames/second: 1055 | Bytes/second: 154k | Errors/minute: 0 Discards/minute: 0 | Total frames: 166k | Total bytes: 44356k Total errors: 0 | Total discards: 0 | Multi/broadcast: 2 Queue overflows: 0 | No buffers: 0 | Frames queued: 0 Buffer coalesces: 854 | MTUs too big: 0 | TSO segments: 0 TSO bytes: 0 | TSO6 segments: 0 | TSO6 bytes: 0 LTM fastp calls: 166k | LTM busy calls: 0 | LTM update calls: 160 LTM yield calls: 0 | LTM drain rss: 0 | LTM total calls: 166k Xon: 0 | Xoff: 0 | Jumbo: 3765 TSO non-TCP drop: 0 | Split hdr drop: 0 | Pktlen: 0 Timeout: 0 | Timeout1: 0 LINK_INFO Current state: up | Up to downs: 6 | Speed: 10000m Duplex: full | Flowcontrol: full RSS Itable updates: 0 | Hash errors: 0 Queue 0 Tx frames: 11780 | Tx bytes: 22306k | Rx frames: 59 Rx bytes: 103k | Requeued: 0 | Hi pri intr: 10735 Queue 1 Tx frames: 1073 | Tx bytes: 143k | Rx frames: 4484 Rx bytes: 33895k | Requeued: 3165 | Hi pri intr: 4160 Queue 2 Tx frames: 306 | Tx bytes: 33404 | Rx frames: 3469 Rx bytes: 3539k | Requeued: 3007 | Hi pri intr: 3831 Queue 3 Tx frames: 153k | Tx bytes: 21872k | Rx frames: 240k Rx bytes: 4184m | Requeued: 801 | Hi pri intr: 143k
7-Mode Flow control recommendations:
Ensure that the flow control interface option is disabled on the storage controller on interfaces using 10g Ethernet.(The host might follow its own best practices, such that the flow control is enabled on the host end using send/receive as needed; however, this is outside of the scope of this article.)
The defined multicast MAC address: 01-80-C2-0-0-01 sends the PAUSE frames, including the pause frame length.
This works at the physical link layer. None of the upper level protocols are considered, which can potentially affect the overall performance of TCP by inducing an artificial delay between router hops, while also having the effect of decreasing the window size due to dropped packets. In large networks, congestion trees can form, which can severely limit the overall network throughput for all attached devices on the network.
It is not recommended to enable flow control throughout the network, including intracluster ports, data switches, and data ports.
Flow control settings by default are set to on for both send and receive flow control for backward compatibility.
Configuring the send and receive flow control to be disabled will need to be set from the ifconfig command and saved to the /etc/rc file to be made persistent.
Example: ifconfig e0a flowcontrol none
Clustered Data ONTAP / ONTAP 9 Flow control recommendations:
It is recommended to match the flow control setting throughout each network path, including intercluster ports, data switches, and data ports. (excluding intracluster /cluster network ports)
A subsequent effort by Cisco, priority flow control (defined in IEEE 802.1Qbb), was intended to replace802.3x PAUSE and provide a mechanism that could be controlled independently for each class of service defined by IEEE802.1p. Instead of pausing all traffic on a network link, priority flow control allows the administrator to selectively pause traffic according to its class of service.
Historically (7-Mode/clustered Data ONTAP), NetApp had recommended that flow control be disabled on all network ports within a NetApp Data ONTAP cluster. This approach is no longer the case. Guidance in this area has since changed, the new recommended best practice is as follows:
- Disable flow control on cluster network ports in the Data ONTAP cluster.
- Flowcontrol on the remaining network ports (the ports that provide data, management, and intercluster connectivity) should be configured to match the settings within the rest of your environment.
Additional Information
N/A