What causes EMS "Nblade.nfsConnResetAndClose with reason OncRpcTcpInvalidRecordMarker"
Applies to
- ONTAP 9
- NFS
Answer
EMS.loglogs an error of typeNblade.invTcpRecordMarker- It will likely be accompanied by an error of type
Nblade.nfsConnResetAndClosewith reasonNblade.InvTcpRecordMarker
Example:
Nblade.invTcpRecordMarker: Invalid TCP record marker received from remote IP address x.x.x.x on local port xxxx. Vserver ID associated with this operation is x.
Invalid reason is TCP_FRAGMENT, invalid record marker is xxxxxxxxx.
Nblade.nfsConnResetAndClose: Shutting down connection with the client. Vserver ID is x; network data protocol is NFS; client IP address:port is x.x.x.x:xxx. local IP address is x.x.x.x; reason is OncRpcTcpInvalidRecordMarker.
- This error message is seen when a TCP connection is reset due to an invalid TCP record marker in an Open Network Computing/Remote Procedure Call request.
- At the head of every NFS call is an RPC call, if this RPC call has the incorrect number of bits or is what the ONTAP would not expect from a client, ONTAP will reset the Client TCP connection.
- That is why the second error
Nblade.nfsConnResetAndClosewill likely be seen with it. - The error message will specify which client is giving the filer the incorrect RPC header, which local TCP port it was using, the Vserver, and the record Marker.
- Below are the different subtypes of invalid TCP record markers for ONC RPC headers.
Note : For all 3 below cases, investigate with the client vendor why the client is sending an invalid record marker.
- TCP_FRAGMENT: The client sent RPC TCP fragments which are not currently supported.
Example:
Nblade.invTcpRecordMarker:error]: Invalid TCP record marker received from remote
IP address x.x.x.x on local port xxxx. Vserver ID associated with this operation is x.
Invalid reason is TCP_FRAGMENT, invalid record marker is 4.
2. TCP_DATA_LENGTH_OVERFLOW: The record marker received was great than the maximum supported size (70KB pre 8.2. 300KB 8.2+, 1MB in 9.0+)
Example:
Nblade.invTcpRecordMarker:error]: Invalid TCP record marker received from remote
IP address x.x.x.x on local port xxxx. Vserver ID associated with this operation is x.
Invalid reason is TCP_DATA_LENGTH_OVERFLOW, invalid record marker is xxxxxxxxx.
3. TCP_DATA_LENGTH_UNDERFLOW: The record marker received was smaller than the minimum supported size (Record marker must be greater than or equal to 40 bytes)
Example:
Nblade.invTcpRecordMarker:error]: Invalid TCP record marker received from remote
IP address x.x.x.x on local port xxxx. Invalid reason is TCP_DATA_LENGTH_UNDERFLOW, invalid record marker is 24.
Additional Information
One possible reason for this message is that the client runs telnet xxx.xxx.xxx.xxx 2049 to confirm the accessibility of NFS.
Linux client :
[root@RHELClient8 ~]# telnet 10.xx.yy.211 2049
Trying 10.xx.yy.211...
Connected to 10.xx.yy.211.
Escape character is '^]'.
^]
telnet> ^C
Connection closed by foreign host.
[root@RHELClient8 ~]# ifconfig -a
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.zz.xx.79 netmask 255.255.255.0 broadcast 10.zz.xx.255
cdot_vsim8::> date ; event log show -event *Nblade*
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
4/14/2021 16:55:48 node1 ERROR Nblade.nfsConnResetAndClose: Shutting down connection with the client. Vserver ID is 2; network data protocol is NFS, Rpc Xid 0x0; client IP address:port is 10.zz.xx.79:33504. local IP address is 10.xx.yy.211; reason is OncRpcTcpInvalidRecordMarker.
4/14/2021 16:55:48 node1 ERROR Nblade.invTcpRecordMarker: Invalid TCP record marker received from remote IP address 10.zz.xx.79 on local port 2049. Vserver ID associated with this operation is 2. Invalid reason is TCP_FRAGMENT, invalid record marker is 218824696.