Many API calls returns with failed errors: 502 Proxy Error
Applies to
- ONTAP 9
- REST API
- Astra Trident
- ONTAP System Manager (OSM)
Issue
- ONTAP System Manager is not reachable on cluster management LIF or the node management LIF where the cluster management sits
- The browser just shows loading state and does nothing
- Many API calls returns with failed errors:
502 Proxy Error
:
AUDIT-MLOG-TXT.GZ
Wed Jan 01 2023 00:00:00 +02:00 [kern_audit:info:11111] 1234e8000001f000 :: Cluster1:ontapi :: 192.168.0.1:11111 :: Cluster1:admin :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 502 Proxy Error
EMS-LOG-FILE.GZ
Wed Jan 01 10:00:00 +0200 [Cluster1-01: mgwd: kern.vm.mmap.return:notice]: mmap(2) by mgwd (pid 2913) for size 5619712 failed: VMEM limit exceeded, limit 4294967296, error 12.
APACHE-ERROR.GZ
[Wed Jan 01 10:00:00.111111 2023 +0000] [proxy:error] [pid 95609:tid 34402477568] [client 192.168.0.1:54858] AH00898: Error reading from remote server returned by //servlets/netapp.servlets.admin.XMLrequest_filer
[Wed Jan 01 10:00:00.111112 2023 +0000] [proxy_ http:error] [pid 95609:tid 34402477568] (20014)Internal error (specific information not available): [client 192.168.0.1:54858] AH01102: error reading status line from remote server 127.0.0.1:936
MGWD.GZ
Wed Jan 01 10:00:00 +02:00 [kern_mgwd:info:3153] A [src/rdb/sql_local_unit.cc 806 (0x823d02000)]: sqlite_logger: SQLite Error: (1) statement aborts at 4: [ATTACH DATABASE '/mroot/etc/cluster_config/rdb/VLDB/_sql/rdb.db' AS vldb;] database vldb is already in use
Wed Jan 01 10:00:00 +02:00 [kern_mgwd:info:3153] 0x823d02000: 8803eb00002a49c4: ERR: SQL_CONTEXT: _execute_setup_stmts:src/sql_context.cc:1056 _execute_setup_stmts: failed to exec statement 'ATTACH DATABASE '/mroot/etc/cluster_config/rdb/VLDB/_sql/rdb.db' AS vldb;' for on demand connection 0x834cabd08: SQL logic error database vldb is already in use
Wed Jan 01 10:00:00 +02:00 [kern_mgwd:info:3153] sqlite exec message: database vldb is already in use
Wed Jan 01 10:00:00 +02:00 [kern_mgwd:info:3153] A [src/rdb/sql_local_unit.cc 806 (0x821a39c00)]: sqlite_logger: SQLite Error: (1) statement aborts at 4: [ATTACH DATABASE '/mroot/etc/cluster_config/rdb/Bcom/_sql/rdb.db' AS bcomd;] database bcomd is already in use
Wed Jan 01 10:00:00 +02:00 [kern_mgwd:info:3153] 0x821a39c00: 8503e80002080bc0: ERR: SQL_CONTEXT: _execute_setup_stmts:src/sql_context.cc:1056 _execute_setup_stmts: failed to exec statement 'ATTACH DATABASE '/mroot/etc/cluster_config/rdb/Bcom/_sql/rdb.db' AS bcomd;' for on demand connection 0x834cabd08: SQL logic error database bcomd is already in use
Wed Jan 01 10:00:00 +02:00 [kern_mgwd:info:3153] sqlite exec message: database bcomd is already in use
- The following errors can also be observed in Trident due to REST API calls failing:
time="2024-07-05T10:06:14Z" level=info msg="deferred syncing TridentBackendConfig 'trident-system/backend-ontap-nas', requeuing; reconcile deferred; problem initializing storage driver 'ontap-nas': error initializing ontap-nas driver: could not create Data ONTAP API client: error creating ONTAP API client: error reading SVM details: API status: failed, Reason: The proxy server failed to communicate with the ontapi server at address 192.168.x.x, Code: 13114" crdControllerEvent=add logLayer=crd_frontend logSource=trident-crd-controller requestID=4d2ea614-0fe1-45c9-95c6-xxxxxxxxxxxx requestSource=CRD workflow="cr=reconcile"