NAS service outage due to exhaustion of lock manager objects
Applies to
ONTAP 9
Issue
- CIFS access fails and/or NFS clients access hangs.
- A packet trace shows NFS open requests being answered with
NFS4ERR_DELAY
- A packet trace shows NFS open requests being answered with
NFS V4 Call (Reply In 18) OPEN DH: 0xfbfc02ab/x_file.bin
NFS V4 Reply (Call In 11) OPEN Status: NFS4ERR_DELAY
- CIFS create requests being answered with
STATUS_INSUFFICIENT_RESOURCES
.
SMB2 Create Request File: x123.p123.local\share1\archive.ppt
SMB2 Create Response, Error: STATUS_INSUFFICIENT_RESOURCES
- Messages similar to the following appear in EMS logs:
ERROR lmgr.owners.limit.hit: The number of owner objects 413800 has reached the limit 413800 for the pool <nodename>. The current number/limit of other lock manager objects are: files 410381/413800 hosts 16/50000, locks 416661/827600.
- Spinhi-stats similar to the following will report
err [SPINNP_ERR_NOLOCK = 77, // No locks available] :
Open: count=106261335 hipri=0 errs=4698192 elapsed: max=5803357.24 avg=831.40
spinhi cpu: max=2965.05 avg=21.47
wafl cpu: max=5803328.35 avg=801.38 restarts=13314078
errs: 77[320325]
Active IQ System Risk Detection
For customers who have enabled AutoSupport™ on their storage systems, the Active IQ Portal provides detailed System Risk reports at the customer and site and system levels. The reports show systems that have specific risks as well as severity levels and mitigation action plans. You may be reading this article as a result of one of those alerts. Autosupports may have shown symptoms specified in this article. Please evaluate the node reporting lmgr.owners.limit.hit
EMS message.
The most common reason for hitting this limit is due to a misbehaving client.
If further assistance is required, please contact NetApp Technical Support