Network latency in a Fibre Channel SAN environment
Applies to
- NetApp FAS/AFF systems
- Fibre Channel Protocol (FCP)
- VMware ESXi
- Linux
- Windows
Issue
- Workloads on hosts accessing LUNs via FCP incur significant application and client latency
- Changes in network infrastructure like switch replacement may lead to clients facing latency
- ONTAP has low latency (less than 5 milliseconds) in latency counters
- In SAN environments when the storage gets a request from a host, an internal timer starts and tracks how long the operations takes in each of the storage delay centers, all of which are internal except for "Network"
- The timer does not stop until the request has been serviced by storage and receives an acknowledgement from the host
- This delay incurred while waiting for host acknowledgement is observed as "Network" latency and hence considered external to storage controller
- Below errors may be observed in EMS logs for FC ports but will not show up in all cases.
Thu Oct 19 07:08:40 +0100 [NetApp-01: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:0e AEN 0x8048 (RECV_ERROR) MboxStatus1 0xfffe MboxStatus2 0x2
Thu Oct 19 08:16:30 +0100 [NetApp-01: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:0e AEN 0x8048 (RECV_ERROR) MboxStatus1 0xfffe MboxStatus2 0x1
Thu Oct 19 08:55:45 +0100 [NetApp-01: fct_tpd_work_thread_0: fcp.io.status:debug]: STIO Adapter:0f AEN 0x8048 (RECV_ERROR) MboxStatus1 0xfffe MboxStatus2 0x1
- Network latency in SAN environment is seen in the
Network
delay center orNetwork
in Active IQ Unified Manager (AIQUM) but not inNetwork Processing
under the Cluster Components volume view and some times it may not report network latency too.
Example:
qos statistics volume latency show
shows high latency in the Network
column for three SAN workloads
cluster::> qos statistics volume latency show -node san_node1
Workload ID Latency Network Cluster Data Disk QoS NVRAM Cloud ----------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- volume_SAN1 17882 20.51ms 20.51ms 0ms 0ms 0ms 0ms 0ms 0ms volume_SAN2 5232 15.00ms 15.00ms 0ms 0ms 0ms 0ms 0ms 0ms volume_SAN3 17160 16.00ms 16.00ms 0ms 0ms 0ms 0ms 0ms 0ms