Skip to main content
NetApp Knowledge Base

Acquisition failed for all clusters in AIQUM due to maximum connection limit exhaustion

  1. Last updated
  2. Save as PDF
Views:
1,529
Visibility:
Public
Votes:
0
Category:
active-iq-unified-manager
Specialty:
om
Last Updated:

Applies to

  • ActiveIQ Unified Manager (AIQUM) 9.6+
  • All OS platforms
  • ONTAP 9.x

Issue

  • Intermittently acquisition is failing for all clusters added to AIQUM
  • Cluster Monitoring Failed and Cluster Not Reachable alerts are triggered by AIQUM
  • However, the acquisition starts working automatically after sometime or if triggered manually.
  • All the prerequisites like AV exclusions and resources availability in terms of CPU/Memory/Disk space are applied on the AIQUM.
  • SSL certificates for AIQUM as well as the ONTAP clusters are valid.
  • AIQUM au.log:
ERROR [common-pool-2064] c.o.s.a.d.n.NetAppOCIEArchivePerformancePackage (NetAppOCIEArchivePerformancePackage.java:381) - Failed to get archive file names from zapi. java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.waitForConnect(Native Method) ~[?:?]
...
Wrapped by: com.onaro.sanscreen.acquisition.framework.datasource.DataSourceErrorException: Failed to connect to <cluster IP/Hostname>
at com.onaro.sanscreen.acquisition.datasource.netapp_ocie.transport.zapi.ZAPIConnection.createDefaultNaServer(ZAPIConnection.java:803) ~[au-datasource-netappfoundation.jar:9.13.0-2023.09.J299]
...

ERROR [common-pool-2064] c.o.s.a.f.d.BaseDataSource (DataSourceErrorException.java:246) - <cluster_IP/Hostname> [Error connecting] - Failed to connect to <cluster IP/Hostname> (connect timed out)

  • AIQUM ocumserver.log shows:
ERROR [oncommand] [reconciliation-0] [c.n.d.c.ClusterStatusListener] Socket connection error for cluster: <cluster IP/Hostname> java.net.ConnectException: Connection timed out: connect
ERROR [oncommand] [reconciliation-0] [c.n.d.c.ClusterStatusListener] Cluster : <cluster IP/Hostname> is not reachable. Generating cluster not reachable event.

  • apache_error.log shows HTTP connection limit has been reached:

[mpm_event:warn] [pid 7215:tid 34401862144] A keepalive connection from ipspace ID -1, remote address <AIQUM IP/Hostname> is being suspended between requests while the 80-connection limit has been reached. (80 active, 8 waiting) Clients should limit the number of concurrent keepalive connections to avoid large performance penalties and/or failures.

[mpm_event:notice] [pid 7215:tid 34402611200] Holding a connection from ipspace ID -1, remote address <AIQUM_IP/Hostname> while 54 others are held and 80 are active
[mpm_event:notice] [pid 7215:tid 34402611200] Holding a connection from ipspace ID -1, remote address <AIQUM_IP/Hostname> while 55 others are held and 80 are active

  • apache_access.log shows status 408 (Request Timed out) for AIQUM API call requests:

<AIQUM IP/Hostname> pii_encrypt/3haVFUKxlfQdtYhedGIaWKrSBVCn+5sImuFntsUoOAk=/pii_encrypt - - [Date/Time] "-" 408 - 38 - 0 - - -
<AIQUM IP/Hostname> pii_encrypt/3haVFUKxlfQdtYhedGIaWKrSBVCn+5sImuFntsUoOAk=/pii_encrypt - - [Date/Time] "-" 408 - 31 - 0 - - -

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.