Skip to main content
NetApp Knowledge Base

When is in-order delivery (IOD) required and how is it set?

  1. Last updated
  2. Save as PDF
Views:
3,426
Visibility:
Public
Votes:
9
Category:
ontap-9
Specialty:
san
Last Updated:

​ ​ ​  

Applies to

  • SAN
  • FlexPod
  • SolidFire All Flash Arrays
  • E-Series All Arrays
  • Netapp Element software
  • Clustered Data ONTAP 8
  • Data ONTAP 7 and earlier

Answer

This Article does not apply to MetroCluster backend configurations. For informations and supported configurations supporting OOD in MetroCluster please review the MetroCluster Installation and Configuration Guide.

What is In-Order Delivery (IOD)?

The order of delivery of frames is maintained within a switch, and is determined by the routing policy in effect. The frame delivery behaviors for each routing policy are:

  • Port-based routing
    All frames received on an incoming port destined for a destination domain are guaranteed to exit the switch in the same order in which they were received.
  • Exchange-based routing
    All frames received on an incoming port for a given exchange are guaranteed to exit the switch in the same order in which they were received. As different paths are chosen for different exchanges, this policy does not maintain the order of frames across exchanges.
    If one switch in the fabric delivers out-of-order exchanges, the exchanges are delivered to the target out-of-order, regardless of the policy configured on other switches in the fabric.
    In a stable fabric, frames are always delivered in order, even when the traffic between switches is shared among multiple paths. However, when topology changes occur in the fabric (for example, if a link goes down), traffic is rerouted around the failure, and some frames could be delivered out of order. Most destination devices tolerate out-of-order delivery, but some do not. By default, out-of-order frame-based delivery is allowed to minimize the number of frames dropped. Enabling in-order delivery (IOD) guarantees that the frames are either delivered in order or dropped. In-order frame delivery can only be forced across topology changes if the fabric contains destination devices that cannot tolerate occasional out-of-order frame delivery.

How does IOD work?

To understand IOD in more detail, we need to discuss a few concepts:

  • Fibre Channel uses Fabric Shortest Path First (FSPF) to determine path availability. During a topology route change, it is possible that the new path is faster or less congested then the previous path. This can lead to out-of-order delivery for frames. Keep in mind that link aggregation changes can also cause new routes to be selected. Path changes may also depend on settings like Dynamic Load Sharing (DLS).
  • A Fibre Channel conversation between the originator and the destination is called a session. Sessions are made up of Frames-->Sequences->Exchanges. Another way to think of these parts is:
    • frame ~ word
    • sequence ~ sentence
    • exchange ~ conversation
      Here is a visual example to illustrate the point:
      1004981-1.jpg
    • Exchange Definition: An exchange describes the operation request and response between N_ports (nodes). The parts of an exchange include:
      • Originator Exchange ID (OX_ID), the requesting N_port
      • Responder Exchange ID (RX_ID), the responding N_port
        A single exchange exists between any pair of N_ports
        Multiple exchanges may exist simultaneously between different pairs of N_ports

For more details on how frames, sequences, and exchanges work together, review the material located here: Fabric Shortest Path First (FSPF) - A link state frame routing protocol that supports load balancing of traffic between equal-cost paths.

Three different mechanisms can be used by FSPF for load balancing:

  1. Round Robin
  2. Exchange based (OXID) - Source ID, Destination ID and Exchange ID
    • In a stable fabric, frames are delivered in order within the exchange
  3. Flow based / Port based - Source ID and Destination ID (Can be the FCID or physical port)
    • In a stable fabric, frames are delivered in order within a single exchange. Two exchanges could use different paths, resulting in the potential for out of order frames.

When IOD is turned on, the following applies to the fabric:

  • Requires FSPF to use Flow based / Port based load balancing
  • Frames using the old path are delivered before new frames are accepted
  • A new route will NOT be added until the hold down period is met on the old path â?? hold down is equal to the E_D_TOV = 2000 ms (default)
  • Frames that are delivered through the old path within the switch latency drop period are dropped
    • Drop period for Cisco default is 500 milliseconds
    • Drop period Brocade default is 650 milliseconds
  • The new frames are delivered through the new path after the switch latency drop period has elapsed

Key Facts & Default settings:

  • By default, load balancing is based on exchange ID on all Cisco switches and 4 Gbps Brocade switches when using different, equal cost paths.
  • Brocade 2 Gbps switches and below only capable of Source / Destination ID routing without link aggregation
  • The IOD setting is turned off by default on Cisco and Brocade switches. However, even without this setting enabled IOD is guaranteed on a stable fabric within an exchange.

When is IOD required?

IOD is required for FCVI in a V-Series MetroCluster or SnapMirror over FC (SMoFC) environment with multiple interswtich links (ISLs) in the fabric where FC-VI traffic can be carried via multiple ISL's (no single ISL dedicated to FC-VI). Exceptions can be found in the MetroCluster Install and Configuration Guide.
New V-Series environments requiring dedicated backend switches do not require these settings.
IOD is also required with NetApp SolidFire storage clusters that use Fibre Channel nodes.

How do you enable IOD on a switch fabric?

  • Set load balancing scheme to src-dst-id  (AptPolicy 1 | 2) 
  • Requires IOD guarantee to be turned ON.
    • Brocade: iodset (iodShow, iodReset)
    • Cisco: disable within VSAN or globally
  • Brocade requires dynamic load balancing within links in a link aggregate to be disabled (dlsReset, dlsSet, dlsShow). This does not apply to Cisco.

What symptoms would I see if frames are being delivered out of order?

The QLogic FCVI adapter reset error messages are seen on the filer console.

Error: Qlogic VI FC Adapter: ISP_CS_VI_ERROR vinum = 0x3 state = 0x3 code = 0x6

Note: This message with error code = 0x6 can be generated if a frame is …

  1. received out of order
  2. discarded in the fabric
  3. discarded at the FC-VI card

Only (1) is a true out-of-order condition. Frame which are discarded as mentioned in (2) and (3) have no relation to IOD settings.

These settings do not apply to and are not tested in a Fabric MetroCluster environment (V-Series and FAS) that requires dedicated backend switches. Out of order delivery in these environments should not occur if the switches are configured according to the “Switch Configuration for Fabric MetroCluster” which is posted on the NOW site.

Determining whether IOD is set

Viewing and changing this parameter:

  1. Enter the iodShow command to view the current IOD setting.

    One of the following messages will be displayed:
    • IOD is set - Enables the in-order delivery (IOD) option. This enforces in-order delivery of frames during a fabric topology change.
    • IOD is not set - Turns off the in-order delivery (IOD) option. This command may cause out-of-order delivery of frames during fabric topology changes.
       
  2. Enter the iodSet command to enable In-Order Delivery.
  3. Enter the iodReset command to disable In-Order Delivery.

    Example:

    switch:admin> iodshow
    IOD is set
    switch:admin> configshow "route.d"
    route.delayReroute:1
    switch:admin> iodreset
    Committing configuration...done.
    switch:admin> iodshow
    IOD is not set
    switch:admin> configshow "route.d"
    route.delayReroute:0

Additional Information

N/A

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.