ONTAP 9 web services remain "offline" status after restarting HTTP processes
Applies to
- ONTAP 9
- ONTAP System Manager
Issue
- The web services are in "offline" status
- Restarting the services have no effect
- Attempting to start an ONTAP upgrade from the CLI shows the following error and system web services are offline at least on some of the nodes:
::> cluster image update -version 9.13.1P13
Starting validation for this update...
Error: show failed: Failed to validate the cluster for image upgrade.
Validation cannot be performed on the following nodes because the HTTP
daemon is offline: Cluster1-03, Cluster1-04.
Ensure the HTTP daemon is online on all nodes and retry the operation.
Run the "system services web node show" command for more information
regarding HTTPD daemon status.
Error: command failed: An error occurred during validation.
::> system services web node show
HTTP HTTP HTTPS Total Total
Node External Enabled Port Port Status HTTP Requests Bytes Served
----------- -------- -------- ------- ----- ------ ------------- ------------
Cluster1-01 true false 80 443 online 56150322 3369517169
Cluster1-02 true false 80 443 online 19071 39543509
Cluster1-03 true false 80 443 offline - -
Cluster1-04 true false 80 443 offline - -
- System Manager and SPI access both fail
apache_error log output:
[Mon Jul 03 15:09:18.956096 2023] [dot:notice] [pid 94700:tid 34378649600] Refreshed the cache of the vserver config file /mroot/etc/www/wsdb/ httpd-vserver.conf, modified at 1688061154
[Mon Jul 03 15:09:18.956478 2023] [dot:notice] [pid 94700:tid 34378649600] Creating http listener on port 65508 for web service FW_BMC
[Mon Jul 03 15:09:18.956920 2023] [dot:notice] [pid 94700:tid 34378649600] Creating https listener on port 65509 for web service FW_BMC
httpd: Syntax error on line 68 of /etc/apache2/ httpd.conf: Syntax error on line 39 of /var/etc/ httpd-custom.conf: Failed to find web service fud: No such file or directory
[Mon Jul 03 16:09:20.003131 2023] [dot:notice] [pid 10464:tid 34378649600] The web services database has been initialized.
OR
- The web services are not in "offline" status
- SPI access does work but System Manager is not
- System Manager is unable to complete some operations and asks that the operation be tried again
- Restarting the services has no effect
- Check for the following in the indicated logs:
apache_error log:
[Wed Dec 13 18:43:44.735012 2023 0000] [dot:error] [pid 11838:tid 34402468608] [client :] [vserver ID 4294912345] [service security] HTTP POST method not allowed
audit-mlog:
00000017.0006bf6c 00c4a9a3 Wed Dec 13 2023 18:43:44 00:00 [kern_audit:info:11838] 8503e80000001df9 :: :http :: : :: :admin :: POST /security/login HTTP/1.1 :: Error: 405 Method Not Allowed
EMS:
[node01-02: spmd: spm.httpd.process.exit:alert]: The web server (httpd) with ID 7134 exited as a result of normal exit (1). This server will attempt to restart
