Skip to main content
NetApp Knowledge Base

Lower than expected iSCSI throughput

Views:
281
Visibility:
Public
Votes:
1
Category:
ontap-9
Specialty:
san
Last Updated:

Applies to

  • Cisco Unified Computing System (UCS)
  • iSCSI
  • MPIO

Issue

  • Observing ~100 MB/s (LVM) to 200 (Single LUN) MB/s instead of expected ~1GB/s throughput in application and verified for application agnostic occurrence with performance testing tool fio after LUN migration to new storage system in an LVM (striping) configuration.
  • LVM
fio -thread -rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test1/file1
READ: bw=93.7MiB/s (98.3MB/s), 93.7MiB/s-93.7MiB/s (98.3MB/s-98.3MB/s), io=10.0GiB (10.7GB), run=109260-109260msec
WRITE: bw=93.7MiB/s (98.2MB/s), 93.7MiB/s-93.7MiB/s (98.2MB/s-98.2MB/s), io=9.00GiB (10.7GB), run=109260-109260msec
 
  • Single LUN
fio -thread-rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test2/file1
READ: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=10.0GiB (10.7GB), run=49574-49574msec
WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=9.00GiB (10.7GB), run=49574-49574msec

 
 
  • Affected configuration (multipath -ll ):
xxxxxxxxxxxxx dm-8 NETAPP ,LUN C-Mode
size=400G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 4:0:0:5 sdab 65:176 active ready running
| |- 2:0:0:5 sdac 65:192 active ready running
| |- 7:0:0:5 sdad 65:208 active ready running
| `- 5:0:0:5 sdar 66:176 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 0:0:0:5 sdf 8:80 active ready running
|- 1:0:0:5 sdl 8:176 active ready running
|- 3:0:0:5 sdam 66:96 active ready running
`- 6:0:0:5 sdav 66:240 active ready running
 
  • When tracing, the following tcpdump filters capture drops:
    • tcpdump:

# tcpdump -i ibft0 -w [path]ibft0.pcap

  • Selective Acknowledgements:

tcp.options.sack.count > 0

  • Duplicate Acknowledgements, (Fast) Retransmissions, Out-Of-Order frames:
tcp.analysis.duplicate_ack || tcp.analysis.retransmission || tcp.analysis.fast_retransmission || tcp.analysis.out_of_order
  • On UCS CLI (nxos) the command show queuing interface ethernet x/xx (affected interfaces for involved LUNs respectively) indicates inbound drops, which indicates NIC buffer exhaustion:
ucs-sas-A(nxos)# sh queuing interface ethernet 1/14

Ethernet1/14 queuing information:
[...]
RX Queuing
[...]

    qos-group 4
    q-size: 29760, HW MTU: 9216 (9216 configured)
    drop-type: drop, xon: 0, xoff: 29760
    Statistics:
        Pkts received over the port             : 16504320
        Ucast pkts sent to the cross-bar        : 16453589
        Mcast pkts sent to the cross-bar        : 0
        Ucast pkts received from the cross-bar  : 10259502
        Pkts sent to the port                   : 10259502
        Pkts discarded on ingress               : 50731
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)
 
    qos-group 5
    q-size: 29760, HW MTU: 9216 (9216 configured)
    drop-type: drop, xon: 0, xoff: 29760
    Statistics:
        Pkts received over the port             : 50303345
        Ucast pkts sent to the cross-bar        : 49818733
        Mcast pkts sent to the cross-bar        : 0
        Ucast pkts received from the cross-bar  : 2162905
        Pkts sent to the port                   : 2162905
        Pkts discarded on ingress               : 484613
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.