Skip to main content
NetApp Knowledge Base

What causes EMS "Nblade.nfsConnResetAndClose with reason OncRpcTcpInvalidRecordMarker"

Views:
5,812
Visibility:
Public
Votes:
3
Category:
ontap-9
Specialty:
nas
Last Updated:

Applies to

  • ONTAP 9
  • NFS

Answer

  • EMS.log logs an error of type Nblade.invTcpRecordMarker
  • It will likely be accompanied by an error of type Nblade.nfsConnResetAndClose with reason Nblade.InvTcpRecordMarker

Example:
Nblade.invTcpRecordMarker: Invalid TCP record marker received from remote IP address x.x.x.x on local port xxxx. Vserver ID associated with this operation is x.
Invalid reason is TCP_FRAGMENT, invalid record marker is xxxxxxxxx.

Nblade.nfsConnResetAndClose: Shutting down connection with the client. Vserver ID is x; network data protocol is NFS; client IP address:port is x.x.x.x:xxx. local IP address is x.x.x.x; reason is OncRpcTcpInvalidRecordMarker.

  • This error message is seen when a TCP connection is reset due to an invalid TCP record marker in an Open Network Computing/Remote Procedure Call request.
  • At the head of every NFS call is an RPC call, if this RPC call has the incorrect number of bits or is what the ONTAP would not expect from a client, ONTAP will reset the Client TCP connection.
  • That is why the second error Nblade.nfsConnResetAndClose will likely be seen with it.
  • The error message will specify which client is giving the filer the incorrect RPC header, which local TCP port it was using, the Vserver, and the record Marker.
  • Below are the different subtypes of invalid TCP record markers for ONC RPC headers.

Note : For all 3 below cases, investigate with the client vendor why the client is sending an invalid record marker.
 

  1. TCP_FRAGMENT: The client sent RPC TCP fragments which are not currently supported.

Example:
Nblade.invTcpRecordMarker:error]: Invalid TCP record marker received from remote
IP address x.x.x.x on local port xxxx. Vserver ID associated with this operation is x.
Invalid reason is TCP_FRAGMENT, invalid record marker is 4.

2. TCP_DATA_LENGTH_OVERFLOW: The record marker received was great than the maximum supported size (70KB pre 8.2. 300KB 8.2+, 1MB in 9.0+)

Example:
Nblade.invTcpRecordMarker:error]: Invalid TCP record marker received from remote
IP address x.x.x.x on local port xxxx. Vserver ID associated with this operation is x.
Invalid reason is TCP_DATA_LENGTH_OVERFLOW, invalid record marker is xxxxxxxxx.

3. TCP_DATA_LENGTH_UNDERFLOW: The record marker received was smaller than the minimum supported size (Record marker must be greater than or equal to 40 bytes)

Example:
Nblade.invTcpRecordMarker:error]: Invalid TCP record marker received from remote
IP address x.x.x.x on local port xxxx. Invalid reason is TCP_DATA_LENGTH_UNDERFLOW, invalid record marker is 24.

Additional Information

One possible reason for this message is that the client runs telnet xxx.xxx.xxx.xxx 2049 to confirm the accessibility of NFS.

Linux client : 
[root@RHELClient8 ~]# telnet 10.xx.yy.211 2049
Trying 10.xx.yy.211...
Connected to 10.xx.yy.211.
Escape character is '^]'.
^]
telnet> ^C
Connection closed by foreign host.

[root@RHELClient8 ~]# ifconfig -a
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.zz.xx.79  netmask 255.255.255.0  broadcast 10.zz.xx.255

cdot_vsim8::> date ; event log show -event *Nblade*

Time                Node             Severity      Event
------------------- ---------------- ------------- ---------------------------
4/14/2021 16:55:48  node1            ERROR         Nblade.nfsConnResetAndClose: Shutting down connection with the client. Vserver ID is 2; network data protocol is NFS, Rpc Xid 0x0; client IP address:port is 10.zz.xx.79:33504. local IP address is 10.xx.yy.211; reason is OncRpcTcpInvalidRecordMarker.
4/14/2021 16:55:48  node1            ERROR         Nblade.invTcpRecordMarker: Invalid TCP record marker received from remote IP address 10.zz.xx.79 on local port 2049. Vserver ID associated with this operation is 2. Invalid reason is TCP_FRAGMENT, invalid record marker is 218824696.

 

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.