Skip to main content
NetApp Knowledgebase

Why is the reported LUN latency higher than the volume latency?

Views:
471
Visibility:
Public
Votes:
1
Category:
fas-systems
Specialty:
perf
Last Updated:

Applies to

  • ONTAP 9
  • clustered ONTAP 8.x
  • R2T (Ready to Transfer)
  • XFER_RDY (Transfer Ready)

Answer

It is often observed that the latency measured for iSCSI (and FCP) is significantly higher than that for the underlying volume, and the operation count at the volume level is higher than that measured on the contained LUNs.

Note: The Volume latency(WAFL latency) is only a subset of the LUN latency, so it’s the expected behavior to see a lower Volume latency compared to the LUN latency, this article focuses on the scenarios where LUN latency is significantly higher than Volume latency

Example:

  • A comparison of the Ops and Latency from the Volume layer and LUN layer for a client having 256KB reads
  • iSCSI protocol as an example, but the same applies to FCP as well

Volume Latency

cluster1::*> statistics show -object lun -instance /vol/vol_linux/lun1 -counter read_data|avg_read_latency|read_ops -raw
Object: lun
Instance: /vol/vol_linux/lun1
Start-time: 10/26/2020 00:55:36
End-time: 10/26/2020 00:55:36
Scope: svm_linux
Number of Constituents: 2 (complete_aggregation)
    Counter                                                     Value
    -------------------------------- --------------------------------
    avg_read_latency                                            215ms
    read_data                                                   85.25MB
    read_ops                                                    333
3 entries were displayed.

LUN latency

cluster1::*> statistics show -object volume -instance vol_linux -counter iscsi_read_data|iscsi_read_latency|iscsi_read_ops -raw

Object: volume
Instance: vol_linux
Start-time: 10/26/2020 01:00:10
End-time: 10/26/2020 01:00:10
Scope: svm_linux

    Counter                                                     Value
    -------------------------------- --------------------------------
    iscsi_read_data                                             85.25MB
    iscsi_read_latency                                          50034us
    iscsi_read_ops                                              1332

Note: Volume Ops is 4 times the LUN Ops, as LUN op size is 256KB, while the Volume Op size is only 64KB

 

The major reason for this significant latency difference is that at the WAFL/Volume layer, operation size is limited to 64KB. If the client has to send an operation with a payload larger than 64KB, the payload has to be broken down into multiple Volume operations. Because of this WAFL side limit, each iSCSI session or FCP login will negotiate settings, specifying the amount of data that can be sent in one single PDU(Protocol Data Units). So the Volume latency only stands for the time it takes to handle one single 64KB PDU, but the LUN latency might be measuring the total handling time of several 64KB PDUs.

LUN latency(iSCSI latency or FCP latency) is measured from when the first PDU of the command is fully received, until the time the last PDU of the response is sent to the output queue of ONTAP. 

7-mode ONTAP

  • In 7-mode ONTAP, this works in a strictly serialized way, an example with a 256KB op size iSCSI write
    • The workflow is as follows:

      1. The client sends out the first 64KB PDU
      2. ONTAP receives the PDU, then after handling the first 64KB write, it sends out an R2T back to the client
      3. The client receives the R2T and then send out the second 64KB PDU
      4. ONTAP receives the PDU, then after handling the second 64KB write, it sends out another R2T back to the client
      5. The client receives the R2T and then send out the third 64KB PDU
      6. ONTAP receives the PDU, then after handling the third 64KB write, it sends out another R2T back to the client
      7. The client receives the R2T and then send out the fourth 64KB PDU
      8. ONTAP receives the PDU, then after handling the fourth 64KB write, it sends out the iSCSI write response to the client, that’s where ONTAP ends the measuring of the LUN latency 
    • Volume latency is only for one single 64KB PDU handling, such as the time spent in step 2, 4, 6 and 8

    • The time spent between each 64KB PDU handling is the Network Round Trip time, which also includes the client handling time, the time that it takes for the client to send out the next 64KB PDU, such as the time between 2 and 4, between 4 and 6, and between 6 and 8

    • LUN latency is for the whole 256KB iSCSI write handling, including 4 64KB PDU handling(Volume latency) plus 3 Network Round Trips

    • FCP has a similar term as iSCSI R2T, which is called XFER_RDY

    • In 7-Mode ONTAP, if the LUN latency is significantly higher than Volume Latency * ROUNDUP(OperationSize / 64KB), it means the R2T(or XFER_RDY for FCP) takes a long time to reach the client, or the client takes a long time to send out the next PDU. It indicates that the data path between the clients and the storage is slow. Example:

      • Congestion
      • Low Bandwidth
      • Packet Losses
      • And so on

Clustered Data ONTAP

  • Clustered Data ONTAP has an optimization on this serialized way of handling for both Reads and Writes

    • Up to 16 parallel 64KB PDUs could be handled at the same time, which means for most cases, the LUN latency shouldn’t be impacted by the Network Round Trip

    • It could be run in parallel, but it doesn’t necessarily mean all the PDUs would always run in parallel, for the same 256KB iSCSI write example:

      • The Minimum of the LUN latency equals to the 64KB PDU handling time(Volume latency)

      • The Maximum of the LUN latency equals to the sum of 4 * 64KB PDU handling time(Volume latency) + 3 Network Round Trips

      • LUN latency falls into the range between the Minimum and the Maximum

    • For some old versions of Clustered Data ONTAP, SAN Writes could NOT be run in parallel

 

With all these in mind, for Clustered Data ONTAP, there are two major scenarios where LUN latency is significantly higher than the LUN latency

  • PDUs are not run in parallel
    • This will not cause any performance impact as long as the Volume latency is low
  • The network or the client is slow so it takes a long time for the R2T/XFER_RDY to be acknowledged and for the next PDU to arrive
    • This will cause a performance impact even if the Volume latency is low

Note: In both scenarios, it should NOT be viewed as a storage performance issue as long as the Volume latency is still low

 

How to distinguish between these two scenarios?

qos statistics workload/volume latency show could be used to tell between these two scenarios:

Example:

cluster1::*> qos statistics workload latency show -workload vol_linux-wid22068

Workload            ID    Latency    Network    Cluster       Data       Disk        QoS       NVRAM      Cloud 

---------------    ------ ---------- ---------- ----------    ---------- ---------- ---------- ---------- ----------

Vol_linux-wid22068    -     210ms    8ms         1ms          181ms      18ms        0ms       2ms        0ms    

  • PDUs are not run in parallel
    • No significant latency is from Network
  • The network or the client is slow so it takes a long time for the R2T/XFER_RDY to be acknowledged and for the next PDU to arrive
    • The majority of the high latency is from Network

Note:

  • In the QoS latency breakdown, Network stands for the latency introduced by the external components, such as Network or Clients
  • As long as there is no high latency from Network, there is no need to be concerned with the LUN latency even if it is significantly higher than the underlying Volume latency

Additional Information

additionalInformation_text