TCP Reassembly Queue Overflows Lead to Poor Performance and Possible Application Disruption on 8.2.5P2 7-mode or ONTAP 9.1P16
Applies to
- ONTAP 9.1
- Data ONTAP 8.2 7-Mode
Issue
A security vulnerability fix in the ONTAP networking stack causes Transmission Control Protocol (TCP) performance to degrade and causes ONTAP to send invalid Selective Acknowledgement (SACK) options in the header of TCP packets. These invalid SACK options can expose an issue in some client networking stacks, causing the clients to fail to retransmit packets on normal retransmit timeout intervals. This interaction can cause application outages. For example, this problem may lead to NFS timeouts or SnapMirror failures.
Signature
- On an applicable version (7-mode 8.2.5P2 or ONTAP 9.1P16)
- Refer to KB TCP reassembly queue overflows lead to poor performance and possible application disruption in versions of ONTAP 9.3, 9.4 and 9.5, if on ONTAP 9.3P9-P11, 9.4P3-P7 or 9.5-9.5P2
- TCP slowness
- Possible application disruption/timeouts
- Network is not lossless
- Indications that the TCP reassembly is being overran can be seen via ‘
netstat -s -p tcp’
7-mode 8.2.5P2
Command: netstat -s -p tcp
Counter: <no. packets> discarded because reassembly queue overflow
ONTAP 9.1P16
Commands:
node run -node <node> netstat -s -p tcp
systemshell -node <node> netstat -s -p tcp
Counters:
For node level command:
<no. packets> discarded because reassembly queue overflow
For systemshell level command:
<no. packets> discarded due to memory problems
-
A packet-trace is needed during a problem to confirm if invalid SACK packets are being sent on the affected ONTAP versions. The invalid SACK packets will have an ACK value between one of the SACK left and right edge pairs.
For each SACK range:
If (SACK left edge <= ACK Value < SACK right edge) then an invalid SACK packet is confirmed.