Skip to main content
NetApp Knowledge Base

What are the flow control best practices for Ethernet?

Views:
13,114
Visibility:
Public
Votes:
9
Category:
ontap-9
Specialty:
nas
Last Updated:

 

Applies to

Data ONTAP

Answer

Best Practices:

Follow flow control best practices when experiencing excessive pause frames, xon / xoff frames, or packets being re-queued, as seen in ifstat -av.

- interface  e1a  (0 hours, 0 minutes, 40 seconds) --
RECEIVE
 Frames/second:    1568  | Bytes/second:    27607k | Errors/minute:       0
 Discards/minute:     0  | Total frames:      248k | Total bytes:      4222m
 Total errors:        0  | Total discards:      0  | Multi/broadcast:     2
 No buffers:          0  | Non-primary u/c:     0  | LRO segments:      172k
 LRO bytes:        3929m | L2 terminate:        0  | Tag drop:            0
 Vlan tag drop:       0  | Vlan untag drop:     0  | Vlan forwards:       0
 Vlan broadcasts:     0  | Vlan unicasts:       0  | CRC errors:          0
 Runt frames:         0  | Fragment:            0  | Long frames:         0
 Jabber:              0  | Bus overruns:        0  | Queue drop:          0
 Xon:                 25  | Xoff:                25  | Jumbo:           68353
 No buf hi:           0
TRANSMIT
 Frames/second:    1055  | Bytes/second:      154k | Errors/minute:       0
 Discards/minute:     0  | Total frames:      166k | Total bytes:     44356k
 Total errors:        0  | Total discards:      0  | Multi/broadcast:     2
 Queue overflows:     0  | No buffers:          0  | Frames queued:       0
 Buffer coalesces:  854  | MTUs too big:        0  | TSO segments:        0
 TSO bytes:           0  | TSO6 segments:       0  | TSO6 bytes:          0
 LTM fastp calls:   166k | LTM busy calls:      0  | LTM update calls:  160
 LTM yield calls:     0  | LTM drain rss:       0  | LTM total calls:   166k
 Xon:                 0  | Xoff:                0  | Jumbo:            3765
 TSO non-TCP drop:    0  | Split hdr drop:      0  | Pktlen:              0
 Timeout:             0  | Timeout1:            0
LINK_INFO
 Current state:       up | Up to downs:         6  | Speed:           10000m
 Duplex:            full | Flowcontrol:       full
RSS
 Itable updates:      0  | Hash errors:         0
Queue 0
 Tx frames:       11780  | Tx bytes:        22306k | Rx frames:          59
 Rx bytes:          103k | Requeued:            0  | Hi pri intr:     10735
Queue 1
 Tx frames:        1073  | Tx bytes:          143k | Rx frames:        4484
 Rx bytes:        33895k | Requeued:         3165  | Hi pri intr:      4160
Queue 2
 Tx frames:         306  | Tx bytes:        33404  | Rx frames:        3469
 Rx bytes:         3539k | Requeued:         3007  | Hi pri intr:      3831
Queue 3
 Tx frames:         153k | Tx bytes:        21872k | Rx frames:         240k
 Rx bytes:         4184m | Requeued:          801  | Hi pri intr:       143k
7-Mode Flow control recommendations:

Ensure that the flow control interface option is disabled on the storage controller on interfaces using 10g Ethernet.(The host might follow its own best practices, such that the flow control is enabled on the host end using send/receive as needed; however, this is outside of the scope of this article.)
 

The defined multicast MAC address: 01-80-C2-0-0-01 sends the PAUSE frames, including the pause frame length.

This works at the physical link layer. None of the upper level protocols are considered, which can potentially affect the overall performance of TCP by inducing an artificial delay between router hops, while also having the effect of decreasing the window size due to dropped packets. In large networks, congestion trees can form, which can severely limit the overall network throughput for all attached devices on the network. 

It is not recommended to enable flow control throughout the network, including intracluster ports, data switches, and data ports. 

Flow control settings by default are set to on for both send and receive flow control for backward compatibility. 

Configuring the send and receive flow control to be disabled will need to be set from the ifconfig command and saved to the /etc/rc file to be made persistent.

Example: ifconfig e0a flowcontrol none

Clustered Data ONTAP / ONTAP 9 Flow control recommendations:

It is recommended to match the flow control setting throughout each network path, including intercluster ports, data switches, and data ports. (excluding intracluster /cluster network ports)

A subsequent effort by Cisco, priority flow control (defined in IEEE 802.1Qbb), was intended to replace802.3x PAUSE and provide a mechanism that could be controlled independently for each class of service defined by IEEE802.1p.  Instead of pausing all traffic on a network link, priority flow control allows the administrator to selectively pause traffic according to its class of service.

Historically (7-Mode/clustered Data ONTAP), NetApp had recommended that flow control be disabled on all network ports within a NetApp Data ONTAP cluster. This approach is no longer the case. Guidance in this area has since changed, the new recommended best practice is as follows:

  • Disable flow control on cluster network ports in the Data ONTAP cluster.
  • Flowcontrol on the remaining network ports (the ports that provide data, management, and intercluster connectivity) should be configured to match the settings within the rest of your environment.

Additional Information

N/A

 

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.