Lower than expected iSCSI throughput
Applies to
- Cisco Unified Computing System (UCS)
- iSCSI
- MPIO
Issue
- Observing ~100 MB/s (LVM) to 200 (Single LUN) MB/s instead of expected ~1GB/s throughput in application and verified for application agnostic occurrence with performance testing tool fio after LUN migration to new storage system in an LVM (striping) configuration.
- LVM
fio -thread -rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test1/file1
READ: bw=93.7MiB/s (98.3MB/s), 93.7MiB/s-93.7MiB/s (98.3MB/s-98.3MB/s), io=10.0GiB (10.7GB), run=109260-109260msec
WRITE: bw=93.7MiB/s (98.2MB/s), 93.7MiB/s-93.7MiB/s (98.2MB/s-98.2MB/s), io=9.00GiB (10.7GB), run=109260-109260msec- Single LUN
fio -thread-rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test2/file1
READ: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=10.0GiB (10.7GB), run=49574-49574msec
WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=9.00GiB (10.7GB), run=49574-49574msec- Affected configuration (
multipath -ll):
xxxxxxxxxxxxx dm-8 NETAPP ,LUN C-Modesize=400G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw|-+- policy='service-time 0' prio=50 status=active| |- 4:0:0:5 sdab 65:176 active ready running| |- 2:0:0:5 sdac 65:192 active ready running| |- 7:0:0:5 sdad 65:208 active ready running| `- 5:0:0:5 sdar 66:176 active ready running`-+- policy='service-time 0' prio=10 status=enabled|- 0:0:0:5 sdf 8:80 active ready running|- 1:0:0:5 sdl 8:176 active ready running|- 3:0:0:5 sdam 66:96 active ready running`- 6:0:0:5 sdav 66:240 active ready running- When tracing, the following tcpdump filters capture drops:
- tcpdump:
# tcpdump -i ibft0 -w [path]ibft0.pcap
- Selective Acknowledgements:
tcp.options.sack.count > 0
- Duplicate Acknowledgements, (Fast) Retransmissions, Out-Of-Order frames:
tcp.analysis.duplicate_ack || tcp.analysis.retransmission || tcp.analysis.fast_retransmission || tcp.analysis.out_of_order- On UCS CLI (nxos) the command show queuing interface ethernet x/xx (affected interfaces for involved LUNs respectively) indicates inbound drops, which indicates NIC buffer exhaustion:
ucs-sas-A(nxos)# sh queuing interface ethernet 1/14Ethernet1/14 queuing information:
[...]
RX Queuing
[...]
qos-group 4
q-size: 29760, HW MTU: 9216 (9216 configured)
drop-type: drop, xon: 0, xoff: 29760
Statistics:
Pkts received over the port : 16504320
Ucast pkts sent to the cross-bar : 16453589
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 10259502
Pkts sent to the port : 10259502
Pkts discarded on ingress : 50731
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
qos-group 5
q-size: 29760, HW MTU: 9216 (9216 configured)
drop-type: drop, xon: 0, xoff: 29760
Statistics:
Pkts received over the port : 50303345
Ucast pkts sent to the cross-bar : 49818733
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 2162905
Pkts sent to the port : 2162905
Pkts discarded on ingress : 484613
Per-priority-pause status : Rx (Inactive), Tx (Inactive)