What are the flow control best practices for Ethernet?
Applies to
ONTAP 9
Answer
- Follow flow control best practices when experiencing excessive pause frames, xon / xoff frames, or packets being re-queued, as seen in
ifstat -av.
- interface e1a (0 hours, 0 minutes, 40 seconds) -- RECEIVE Frames/second: 1568 | Bytes/second: 27607k | Errors/minute: 0 Discards/minute: 0 | Total frames: 248k | Total bytes: 4222m Total errors: 0 | Total discards: 0 | Multi/broadcast: 2 No buffers: 0 | Non-primary u/c: 0 | LRO segments: 172k LRO bytes: 3929m | L2 terminate: 0 | Tag drop: 0 Vlan tag drop: 0 | Vlan untag drop: 0 | Vlan forwards: 0 Vlan broadcasts: 0 | Vlan unicasts: 0 | CRC errors: 0 Runt frames: 0 | Fragment: 0 | Long frames: 0 Jabber: 0 | Bus overruns: 0 | Queue drop: 0 Xon: 25 | Xoff: 25 | Jumbo: 68353 No buf hi: 0 TRANSMIT Frames/second: 1055 | Bytes/second: 154k | Errors/minute: 0 Discards/minute: 0 | Total frames: 166k | Total bytes: 44356k Total errors: 0 | Total discards: 0 | Multi/broadcast: 2 Queue overflows: 0 | No buffers: 0 | Frames queued: 0 Buffer coalesces: 854 | MTUs too big: 0 | TSO segments: 0 TSO bytes: 0 | TSO6 segments: 0 | TSO6 bytes: 0 LTM fastp calls: 166k | LTM busy calls: 0 | LTM update calls: 160 LTM yield calls: 0 | LTM drain rss: 0 | LTM total calls: 166k Xon: 0 | Xoff: 0 | Jumbo: 3765 TSO non-TCP drop: 0 | Split hdr drop: 0 | Pktlen: 0 Timeout: 0 | Timeout1: 0 LINK_INFO Current state: up | Up to downs: 6 | Speed: 10000m Duplex: full | Flowcontrol: full RSS Itable updates: 0 | Hash errors: 0 Queue 0 Tx frames: 11780 | Tx bytes: 22306k | Rx frames: 59 Rx bytes: 103k | Requeued: 0 | Hi pri intr: 10735 Queue 1 Tx frames: 1073 | Tx bytes: 143k | Rx frames: 4484 Rx bytes: 33895k | Requeued: 3165 | Hi pri intr: 4160 Queue 2 Tx frames: 306 | Tx bytes: 33404 | Rx frames: 3469 Rx bytes: 3539k | Requeued: 3007 | Hi pri intr: 3831 Queue 3 Tx frames: 153k | Tx bytes: 21872k | Rx frames: 240k Rx bytes: 4184m | Requeued: 801 | Hi pri intr: 143k
- It is recommended to match the flow control setting throughout each network path, including intercluster ports, data switches, and data ports. (excluding intracluster /cluster network ports)
- A subsequent effort by Cisco, priority flow control (defined in IEEE 802.1Qbb), was intended to replace802.3x PAUSE and provide a mechanism that could be controlled independently for each class of service defined by IEEE802.1p. Instead of pausing all traffic on a network link, priority flow control allows the administrator to selectively pause traffic according to its class of service.
- Historically (7-Mode/clustered Data ONTAP), NetApp had recommended that flow control be disabled on all network ports within an ONTAP cluster. This approach is no longer the case. Guidance in this area has since changed, the new recommended best practice is as follows:
- Disable flow control on cluster network ports in the ONTAP cluster.
- Flowcontrol on the remaining network ports (the ports that provide data, management, and intercluster connectivity) should be configured to match the settings within the rest of your environment.
Additional Information
N/A