What are the NFSv4 Storepools, why do they exist?
Applies to
- ONTAP 9
- NFSv4
Answer
- ONTAP stores information about the NFSv4+ states for client connections in memory allocations called storepools
- Each storepool is a collection of memory allocations (pools) that are used to facilitate NFSv4/4.1 state work for client connections.
- There are 12 different storepool pairs that encompass a variety of areas:
| StorePool Name | Description |
|---|---|
| storePool_ByteLockAlloc | Byte-range locks |
| storePool_ClientAlloc | ClientIDs |
| storePool_CopyStateAlloc | Used for copy offload related work |
| storePool_DelegAlloc | Delegations for reads and writes (client caching) |
| storePool_DelegStateAlloc | Delegation StateID |
| storePool_LayoutAlloc | Layouts (used for pNFS referrals) |
| storePool_LayoutStateAlloc | Layout StateID |
| storePool_LockStateAlloc | Lock StateID |
| storePool_OpenAlloc | Opens |
| storePool_OpenStateAlloc | Open StateID |
| storePool_OwnerAlloc | Owners |
| storePool_StringAlloc | Stores opaque data sent for ClientID and Owner |
- Where are storePool resources located?
- Each node has its own unique storePool resources.
- These storePool resources are stored in the Network blade (nblade) of each node.
- They are utilized by clients that are mounting LIFs local to the node.
- How do storePool resources differ from lock manager objects?
- The storePool resources are specifically for doing NFSv4/4.1 state work, they do not keep track of open/locked files at the file system layer (dblade).
- Lock manager objects are responsible for the file system open/lock tracking, including resolution of open/lock conflicts.
- How does storePool exhaustion occur differ from lock manager object exhaustion?
- Exhaustion of the storePool occurs at the Network layer (nblade) of the node where the LIF is mounted.
- This is an exhaustion of the NFSv4/4.1 objects for tracking state.
- Lock manager exhaustion is exhaustion of the objects responsible for opening/locking files at the file system layer (Data blade or dblade).
- Lock manager exhaustion would affect the volumes located on the node where exhaustion occurred.
- Monitoring storePool consumption:
- storePool objects can be monitored with
statisticscounters and alert warnings when a specific "pool" becomes full. - The pool allocation numbers are dynamic and should be changing constantly as files are opened and closed by clients.
- NetApp has the capability to manually retrieve the counters via command which could be leveraged via scripting for alerting.
- In ONTAP 9.2 and later, an EMS event exists to alert when a storePool resource reaches 80% of its maximum threshold.
- storePool objects can be monitored with
- How can specific clients potentially cause problems?
- In certain circumstances, clients do not close their OPENs in a way that the ONTAP node is expecting.
- When this occurs, the client is unaware that it still has that OPEN allocated.
- In this case, the server will not remove the OpenState object and the resource is never returned to the pool.
- If this behavior continues, storePool exhaustion can occur as the client behavior orphans resources on the server.
- Dumping nfsv4 locks can show which client is taking up all of the allocations in the storePool.
- Once this client is restarted, ONTAP frees associated storePool resources associated with that client.
- A client mounts a LIF on node A and a LIF on node B via NFSv4. Both of these mounts access a volume on node A. Exhaustion of storePool resource occurs on Node A. Which mount will be affected?
- Only connections on Node A will be affected.
- These allocations are on the "networking" layer (nblade) of ONTAP which is the node that is being used for connections.
- Where the volumes are located is not relevant for the issue.
| Physical Memory Range | Limit Source | Max Compounds | Max Client Objects | Max Owner Objects | Max Open State Objects |
Max Deleg State Objects |
Max Lock State Objects |
Max Layout State Objects |
Max Copy State Objects |
Max Open Objects |
Max Deleg Objects |
Max ByteLock Objects |
Max Layout Objects |
Max LayoutLock Objects |
Max String Objects |
Max Sessions |
Max Execs |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 - 2 GB | Static table | 1,500,000 | 1,000 | 6,000 | 26,000 | 26,000 | 26,000 | 26,000 | 10,000 | 26,000 | 26,000 | 26,000 | 26,000 | 26,000 | 26,000 | 1,000 | 512,000 |
| 2 - 4 GB | Static table | 15,000,000 | 12,000 | 128,000 | 512,000 | 512,000 | 512,000 | 512,000 | 10,000 | 512,000 | 512,000 | 512,000 | 512,000 | 512,000 | 256,000 | 15,000 | 512,000 |
| 4 - 15 GB | Static table | 15,000,000 | 12,000 | 128,000 | 512,000 | 512,000 | 512,000 | 512,000 | 10,000 | 512,000 | 512,000 | 512,000 | 512,000 | 512,000 | 256,000 | 15,000 | 512,000 |
| 15 - 23 GB | Static table | 15,000,000 | 12,000 | 128,000 | 512,000 | 512,000 | 512,000 | 512,000 | 10,000 | 512,000 | 512,000 | 512,000 | 512,000 | 512,000 | 256,000 | 15,000 | 750,000 |
| 23 - 31 GB | Static table | 150,000,000 | 100,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 10,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 100,000 | 1,500,000 |
| 31 - 63 GB | Static table | 150,000,000 | 100,000 | 1,000,000 | 1,000,000 | 1,000,000 | 2,000,000 | 1,000,000 | 10,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 100,000 | 1,500,000 |
| 63 - 127 GB | Static table | 150,000,000 | 100,000 | 1,000,000 | 1,000,000 | 1,000,000 | 4,000,000 | 1,000,000 | 10,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 100,000 | 1,500,000 |
| 127 - 255 GB | Static table | 150,000,000 | 100,000 | 1,000,000 | 1,000,000 | 1,000,000 | 4,000,000 | 1,000,000 | 10,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 1,000,000 | 100,000 | 1,500,000 |
| Above 255 GB (Example: 300 GB) | Per-GB scaling | 150,000,000 | 300,000 | 1,800,000 | 9,000,000 | 6,000,000 | 6,000,000 | 3,000,000 | 3,000,000 | 9,000,000 | 6,000,000 | 7,800,000 | 3,000,000 | 3,000,000 | 3,000,000 | 300,000 | 153,600,000 |
| Above 255 GB (Example: 512 GB) | Per-GB scaling | 150,000,000 | 512,000 | 3,072,000 | 15,360,000 | 10,240,000 | 10,240,000 | 5,120,000 | 5,120,000 | 15,360,000 | 10,240,000 | 13,312,000 | 5,120,000 | 5,120,000 | 5,120,000 | 512,000 | 262,144,000 |
| Above 255 GB (Example: 1 TB) | Per-GB scaling | 150,000,000 | 1,000,000 | 6,000,000 | 30,000,000 | 20,000,000 | 20,000,000 | 10,000,000 | 10,000,000 | 30,000,000 | 20,000,000 | 26,000,000 | 10,000,000 | 10,000,000 | 10,000,000 | 1,000,000 | 512,000,000 |
Additional Information