How to troubleshoot pBlk exhaustion due to Fpolicy Server

Applies to

• Data Ontap 7-mode
• Fpolicy

Description

• Fpolicy is a feature, with both native and via 3rd-party applications, that provides useful add-on features to Data ONTAP. It is generally used to provide file blocking (that is, keep MP3s from my server), HSM (moving infrequently used files to lower tiered storage), Quotas and Auditing. Most of these actions are performed by a 3rd party application. As a result, the response time of an external Fpolicy server directly impacts the ability of a Storage Controller to respond to client requests.
• Since Fpolicy servers are external to the Storage Controller, the usage of pBlks increase as each request from the client accounts for one pBlk and the scan request to the Fpolicy Server accounts for a second pBlk. The sooner the Fpolicy server can complete the scan request, the sooner Data ONTAP can respond to the original client request and free up the pBlk. Sizing of Microsoft based Fpolicy Server deployments are critical for the Storage Controller to function at peak efficiency.
• There are three issues to be considered when looking at pBlk exhaustion and external Fpolicy Servers:
• The number of Fpolicy Servers

The maximum number of Fpolicy requests the Storage Controller can send to an Fpolicy Server at any given time is 50. If 100 requests come in at the same time, one Fpolicy Server will have to process the first 50 requests before it can start the second 50 requests. In this scenario, the Max gOffloadQueue depth would become 50 since the storage controller had to wait for the first block of 50 to finish before sending the second block of 50. In this example, pBlk exhaustion might not have occurred, but it highlights that as more clients are added to a Storage Controller, the need for optimal Fpolicy Server performance becomes necessary.

• The speed of Fpolicy Servers

The speed of external Fpolicy Servers is critical; for this reason it is recommended to run Fpolicy Servers on dedicated hardware rather than running them as Virtual Machines. If the performance of the external Fpolicy Server is degraded, it will take longer to respond to Storage Controller Fpolicy requests, resulting in pBlks being held for longer times. If the speed of the Fpolicy Server is so degraded and enough clients send requests in a short period of time, pBlk Exhaustion can occur.

• The configuration of Fpolicy Servers

Fpolicy vendors control the tunable options for their application. The best place to start is the installation and configuration guides from the Fpolicy Server vendor to make sure that the best practices are being met for the respective product. Configurations not meeting vendor best practices will likely result in decreased performance, putting the Storage Controller at the risk of pBlk exhaustion.

• A sign that your Fpolicy Server could be contributing to pBlk exhaustion is the following:

Wed Sep 22 11:03:59 IST [cifs.server.infoMsg:info]: CIFS: Warning for server \filer1: Connection terminated.
Wed Sep 22 11:04:35 IST [fpolicy.fscreen.enable:info]: FPOLICY: File policy HSM1 (file screening) is enabled.
Wed Sep 22 11:05:06 IST [fpolicy.fscreen.server.droppedConn:warning]: FPOLICY: File policy server 192.168.1.100 for fscreen policy HSM1 has disconnected from the filer.
Wed Sep 22 11:06:10 IST [fpolicy.fscreen.server.pingRejected:error]: FPOLICY: Error trying to get status from file screening server \filer1 for policy HSM1 [0x6d].
Wed Sep 22 11:11:01 IST [ems.engine.inputSuppress:info]: Event 'cifs.stats.pBlkExhaust' suppressed 34 times since Tue Sep 21 17:48:01 IST 2010.
Wed Sep 22 11:11:01 IST [cifs.stats.pBlkExhaust:info]: CIFS: All CIFS control blocks for the STANDARD pool are in use. The request for a new control block can not be granted.