Generate CSR doesn't work in AIQUM

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 24

Visibility:: Public

Votes:: 0

Category:: active-iq-unified-manager

Specialty:: om

Last Updated:

Applies to

Active IQ Unified Manager (AIQUM) 9.x
Red Hat Enterprise Linux (RHEL)
Certificate Signing Request (CSR)

Issue

In AIQUM running on RHEL, navigating to General → HTTPS Certificates → Download HTTPS Certificate Signing Request does nothing — no CSR file is offered for download and the GUI returns no visible error.
Reviewing /opt/netapp/essentials/jboss/server/onaro/log/ocumserver.log at the time of the click shows a BouncyCastle signer failure:

ERROR [admin] [default task-3118] [c.n.o.r.m.e.GlobalExceptionHandler] cannot create signer: Key must not be null
org.bouncycastle.operator.OperatorCreationException: cannot create signer: Key must not be null
at deployment.dfm-app.war//org.bouncycastle.operator.jcajce.JcaContentSignerBuilder.build(Unknown Source)
at deployment.dfm-app.war//com.netapp.dfm.app.common.security.CertificateManagerImpl.createCertificationRequest

(CertificateManagerImpl.java:110)
…
Caused by: java.security.InvalidKeyException: Key must not be null

The same error is visible in the browser HAR capture as a 500 response to the Download HTTPS Certificate Signing Request REST call.
A keytool -list against the JBoss keystore may show an entry created at install/upgrade time with an unexpected key size or signature algorithm (for example a 3072-bit RSA key when the in-product self-signed entry should be 2048-bit)
This indicates the private key material the GUI expects to sign with is missing or mismatched in the keystore.

Cause

This is an environmental issue specific to the customer's host:
SELinux in enforcing mode or other host-hardening can block the AIQUM jboss / umadmin user from reading or rewriting the keystore files that the Generate CSR flow needs.
Security automation tools such as Ansible, CIS-hardening playbooks, or other configuration-management agents periodically reset file ownership, mode, or SELinux labels on AIQUM directories — most importantly on /opt/netapp/essentials/jboss/server/onaro/cert/ and /opt/netapp/data/ocie/.
Along with modified file permission, security hardening automation also restricts the maintenance user from various locations for eg: removing it from /etc/passwd, /etc/shadow etc causing AIQUM GUI login failure via maintenance_user/using restricted comments such as dfm ssl etc
When that happens, the GUI flow can no longer load the private key, BouncyCastle is handed a null key, and the CSR generation fails silently in the UI with the stack trace shown above in ocumserver.log.
This is not a defect in the AIQUM product code path itself — the same GUI flow works on a fresh install where SELinux/Ansible have not yet altered the keystore files.

Solution

Make sure maintenance_user is added as an user in RHEL such as /etc/passwd and /etc/shadow
Access the maintenance_user to access AIQUM GUI and download CSR
In case there are still issues, use the manual keytool option as below:

WARNING

Working with keytool directly against an in-production AIQUM keystore carries risk — always take a VM snapshot and a copy of /opt/netapp/essentials/jboss/server/onaro/cert/ and /opt/netapp/data/ocie/ before making changes.

Goal: confirm the keystore / cert objects are consistent and have valid private key material, then either re-run the supported CLI helper or, as a fall-back, recreate the keystore entry with keytool using a matching algorithm and key length.

Check the signature algorithm and key length of the current keystore entry — the algorithm and key length used to create the keystore entry must match the algorithm and key length expected by the CA when the CSR is signed:

keytool -list -v -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

Note the Signature algorithm name (for example SHA256withRSA) and the Subject Public Key Algorithm / key size (for example 2048-bit RSA key).
Verify the two keystore copies are in sync. The JBoss keystore and the OCIE keystore should be byte-for-byte identical — if they are not, the GUI flow can hit the "Key must not be null" condition:

cksum /opt/netapp/data/ocie/server.keystore
cksum /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore

Both cksum values must match. If they do not, restore from a known good backup or re-sync them from the JBoss copy.
Use the supported cert.sh helper to regenerate the keystore entry:

/opt/netapp/essentials/bin/cert.sh createcustom <FQDN> <email> <Org> <City> <Country> 2048 730 '<SAN list, comma separated>' 'true'

This is the preferred path because it keeps the JBoss and OCIE keystore copies in sync and reuses the AIQUM-managed credential store.
If cert.sh does not fit (for example you need a different alias / algo), fall back to keytool directly. Identify the alias first:

keytool -list -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -v -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

Then generate a key pair (matching the algo/key-length expected by the CA) and a CSR:

keytool -genkeypair -alias keyentry -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storetype JKS -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -dname "CN=<FQDN>" -ext san=dns:<FQDN>,dns:<short-name>,ip:<mgmt-ip> -validity 730 -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

keytool -certreq -alias keyentry -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storetype JKS -sigalg SHA256withRSA -file server.csr -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-)) -keypass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

Get the certificate signed by the CA and confirm the returned certificate has the same signature algorithm and key length as the CSR (a mismatch here will reproduce the same "Key must not be null" symptom on import).
Import the signed certificate back into the keystore:

keytool -importcert -alias keyentry -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storetype JKS -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-)) -trustcacerts -file </path_to_the_cert_file/cert_name>.crt

Alternatively via the supported helper:

/opt/netapp/essentials/bin/cert.sh import </path_to_the_cert_file/cert_name>.crt

Re-sync OCIE and restart AIQUM services so the new keystore is loaded by both JBoss and OCIE.
Address the environmental root cause so the issue does not recur:
- Confirm SELinux state with getenforce and, if it is Enforcing, review the AVC denials in /var/log/audit/audit.log against the AIQUM directories and add the required policy / labels.
- Audit the Ansible / configuration-management playbooks that run against the AIQUM host and exclude the AIQUM-owned paths from generic chmod/chown/restorecon steps.

Partner Notes

partnerNotes_text

Additional Information

Files touched by the Generate CSR GUI flow :
- /opt/netapp/essentials/bin/cert.sh
- /opt/netapp/essentials/bin/certificatemanager/platform-base-display-key.jar
- /opt/netapp/data/ocie/server.keystore
- /opt/netapp/essentials/jboss/standalone/data/jbossCredStore.cs
- /opt/netapp/essentials/bin/elytronstore.sh
- /opt/netapp/essentials/conf/server.properties
- /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore
The two server.keystore files (under /opt/netapp/data/ocie/ and /opt/netapp/essentials/jboss/server/onaro/cert/) must have an identical cksum.

Internal Notes

Source case: 2010649653 (UBS, AIQUM 9.16 on RHEL)
CPE escalation: CPE-12443 — AIQUM 9.16 — Generate CSR from GUI fails: BouncyCastle "cannot create signer: Key must not be null" (UBS, 2010649653)
Bug ID: CAIQUM-7005
Patch: a fixed keystoresetup.jar is available via mysupport — AIQUM 9.18D21
Both server.keystore copies must have the same cksum:

[root@scs001105021 ~]# cksum /opt/netapp/data/ocie/server.keystore
1751053047 9941 /opt/netapp/data/ocie/server.keystore
[root@scs001105021 ~]# cksum /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore
1751053047 9941 /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore

On the source case the customer's Ansible / hardening modified file permissions on the AIQUM paths, and SELinux was in enforcing mode — both contributed
A temporary setenforce 0 during the remote session was used only to confirm the SELinux contribution and was reverted afterwards; long-term fix is to either exclude the AIQUM paths from the hardening playbook or to add the required SELinux policy.