Skip to main content
NetApp Knowledgebase

What are the flow control best practices for Ethernet?

Views:
1,318
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
network
Last Updated:

 

Applies to

Data ONTAP

Answer

Best Practices:

Follow flow control best practices when experiencing excessive pause frames, xon / xoff frames, or packets being re-queued, as seen in ifstat -av.

- interface  e1a  (0 hours, 0 minutes, 40 seconds) --

RECEIVE

 Frames/second:    1568  | Bytes/second:    27607k | Errors/minute:       0

 Discards/minute:     0  | Total frames:      248k | Total bytes:      4222m

 Total errors:        0  | Total discards:      0  | Multi/broadcast:     2

 No buffers:          0  | Non-primary u/c:     0  | LRO segments:      172k

 LRO bytes:        3929m | L2 terminate:        0  | Tag drop:            0

 Vlan tag drop:       0  | Vlan untag drop:     0  | Vlan forwards:       0

 Vlan broadcasts:     0  | Vlan unicasts:       0  | CRC errors:          0

 Runt frames:         0  | Fragment:            0  | Long frames:         0

 Jabber:              0  | Bus overruns:        0  | Queue drop:          0

 Xon:                 25  | Xoff:                25  | Jumbo:           68353

 No buf hi:           0

TRANSMIT

 Frames/second:    1055  | Bytes/second:      154k | Errors/minute:       0

 Discards/minute:     0  | Total frames:      166k | Total bytes:     44356k

 Total errors:        0  | Total discards:      0  | Multi/broadcast:     2

 Queue overflows:     0  | No buffers:          0  | Frames queued:       0

 Buffer coalesces:  854  | MTUs too big:        0  | TSO segments:        0

 TSO bytes:           0  | TSO6 segments:       0  | TSO6 bytes:          0

 LTM fastp calls:   166k | LTM busy calls:      0  | LTM update calls:  160

 LTM yield calls:     0  | LTM drain rss:       0  | LTM total calls:   166k

 Xon:                 0  | Xoff:                0  | Jumbo:            3765

 TSO non-TCP drop:    0  | Split hdr drop:      0  | Pktlen:              0

 Timeout:             0  | Timeout1:            0

LINK_INFO

 Current state:       up | Up to downs:         6  | Speed:           10000m

 Duplex:            full | Flowcontrol:       full

RSS

 Itable updates:      0  | Hash errors:         0

Queue 0

 Tx frames:       11780  | Tx bytes:        22306k | Rx frames:          59

 Rx bytes:          103k | Requeued:            0  | Hi pri intr:     10735

Queue 1

 Tx frames:        1073  | Tx bytes:          143k | Rx frames:        4484

 Rx bytes:        33895k | Requeued:         3165  | Hi pri intr:      4160

Queue 2

 Tx frames:         306  | Tx bytes:        33404  | Rx frames:        3469

 Rx bytes:         3539k | Requeued:         3007  | Hi pri intr:      3831

Queue 3

 Tx frames:         153k | Tx bytes:        21872k | Rx frames:         240k

 Rx bytes:         4184m | Requeued:          801  | Hi pri intr:       143k

7-Mode Flow control recommendations:

Ensure that the flow control interface option is disabled on the storage controller on interfaces using 10g Ethernet.(The host might follow its own best practices, such that the flow control is enabled on the host end using send/receive as needed; however, this is outside of the scope of this article.)
 

The defined multicast MAC address: 01-80-C2-0-0-01 sends the PAUSE frames, including the pause frame length.

This works at the physical link layer. None of the upper level protocols are considered, which can potentially affect the overall performance of TCP by inducing an artificial delay between router hops, while also having the effect of decreasing the window size due to dropped packets. In large networks, congestion trees can form, which can severely limit the overall network throughput for all attached devices on the network. 

It is not recommended to enable flow control throughout the network, including intracluster ports, data switches, and data ports. 

Flow control settings by default are set to on for both send and receive flow control for backward compatibility. 

Configuring the send and receive flow control to be disabled will need to be set from the ifconfig command and saved to the /etc/rc file to be made persistent.

Example: ifconfig e0a flowcontrol none

Clustered Data ONTAP / ONTAP 9 Flow control recommendations:

It is recommended to match the flow control setting throughout each network path, including intercluster ports, data switches, and data ports. (excluding intracluster /cluster network ports)

A subsequent effort by Cisco, priority flow control (defined in IEEE 802.1Qbb), was intended to replace802.3x PAUSE and provide a mechanism that could be controlled independently for each class of service defined by IEEE802.1p.  Instead of pausing all traffic on a network link, priority flow control allows the administrator to selectively pause traffic according to its class of service.

Historically (7-Mode/clustered Data ONTAP), NetApp had recommended that flow control be disabled on all network ports within a NetApp Data ONTAP cluster. This approach is no longer the case. Guidance in this area has since changed, the new recommended best practice is as follows:

  • Disable flow control on cluster network ports in the Data ONTAP cluster.
  • Flowcontrol on the remaining network ports (the ports that provide data, management, and intercluster connectivity) should be configured to match the settings within the rest of your environment.

Additional Information

N/A