Skip to main content
NetApp Knowledge Base

Generate CSR doesn't work in AIQUM

  1. Last updated
  2. Save as PDF
Views:
7
Visibility:
Public
Votes:
0
Category:
active-iq-unified-manager
Specialty:
om
Last Updated:

Applies to

  • Active IQ Unified Manager (AIQUM) 9.x
  • Red Hat Enterprise Linux (RHEL)
  • Certificate Signing Request (CSR)

Issue

  • In AIQUM running on RHEL, navigating to General → HTTPS Certificates → Download HTTPS Certificate Signing Request does nothing — no CSR file is offered for download and the GUI returns no visible error.
  • Reviewing /opt/netapp/essentials/jboss/server/onaro/log/ocumserver.log at the time of the click shows a BouncyCastle signer failure:

ERROR [admin] [default task-3118] [c.n.o.r.m.e.GlobalExceptionHandler] cannot create signer: Key must not be null
org.bouncycastle.operator.OperatorCreationException: cannot create signer: Key must not be null
at deployment.dfm-app.war//org.bouncycastle.operator.jcajce.JcaContentSignerBuilder.build(Unknown Source)
at deployment.dfm-app.war//com.netapp.dfm.app.common.security.CertificateManagerImpl.createCertificationRequest

(CertificateManagerImpl.java:110)
…
Caused by: java.security.InvalidKeyException: Key must not be null

  • The same error is visible in the browser HAR capture as a 500 response to the Download HTTPS Certificate Signing Request REST call.
  • A keytool -list against the JBoss keystore may show an entry created at install/upgrade time with an unexpected key size or signature algorithm (for example a 3072-bit RSA key when the in-product self-signed entry should be 2048-bit)
  • This indicates the private key material the GUI expects to sign with is missing or mismatched in the keystore.

Cause

  • This is an environmental issue specific to the customer's host:
  • SELinux in enforcing mode or other host-hardening can block the AIQUM jboss / umadmin user from reading or rewriting the keystore files that the Generate CSR flow needs.
  • Security automation tools such as Ansible, CIS-hardening playbooks, or other configuration-management agents periodically reset file ownership, mode, or SELinux labels on AIQUM directories — most importantly on /opt/netapp/essentials/jboss/server/onaro/cert/ and /opt/netapp/data/ocie/. When that happens, the GUI flow can no longer load the private key, BouncyCastle is handed a null key, and the CSR generation fails silently in the UI with the stack trace shown above in ocumserver.log.
  • This is not a defect in the AIQUM product code path itself — the same GUI flow works on a fresh install where SELinux/Ansible have not yet altered the keystore files.

Solution

WARNING

  • Working with keytool directly against an in-production AIQUM keystore carries risk — always take a VM snapshot and a copy of /opt/netapp/essentials/jboss/server/onaro/cert/ and /opt/netapp/data/ocie/ before making changes.

 

Goal: confirm the keystore / cert objects are consistent and have valid private key material, then either re-run the supported CLI helper or, as a fall-back, recreate the keystore entry with keytool using a matching algorithm and key length.

  1. Check the signature algorithm and key length of the current keystore entry — the algorithm and key length used to create the keystore entry must match the algorithm and key length expected by the CA when the CSR is signed:

keytool -list -v  -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storepass $(/opt/netapp/essentials/bin/elytronstore.sh  /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

  1. Note the Signature algorithm name (for example SHA256withRSA) and the Subject Public Key Algorithm / key size (for example 2048-bit RSA key).
  2. Verify the two keystore copies are in sync. The JBoss keystore and the OCIE keystore should be byte-for-byte identical — if they are not, the GUI flow can hit the "Key must not be null" condition:

cksum /opt/netapp/data/ocie/server.keystore
cksum /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore

  1. Both cksum values must match. If they do not, restore from a known good backup or re-sync them from the JBoss copy.
  2. Use the supported cert.sh helper to regenerate the keystore entry:

/opt/netapp/essentials/bin/cert.sh createcustom <FQDN> <email> <Org> <City> <Country> 2048 730 '<SAN list, comma separated>' 'true'

  1. This is the preferred path because it keeps the JBoss and OCIE keystore copies in sync and reuses the AIQUM-managed credential store.
  2. If cert.sh does not fit (for example you need a different alias / algo), fall back to keytool directly. Identify the alias first:

keytool -list -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -v -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

  1. Then generate a key pair (matching the algo/key-length expected by the CA) and a CSR:

keytool -genkeypair -alias keyentry -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storetype JKS -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -dname "CN=<FQDN>" -ext san=dns:<FQDN>,dns:<short-name>,ip:<mgmt-ip> -validity 730 -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

keytool -certreq -alias keyentry -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storetype JKS -sigalg SHA256withRSA -file server.csr -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-)) -keypass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-))

  1. Get the certificate signed by the CA and confirm the returned certificate has the same signature algorithm and key length as the CSR (a mismatch here will reproduce the same "Key must not be null" symptom on import).
  2. Import the signed certificate back into the keystore:

keytool -importcert -alias keyentry -keystore /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore -storetype JKS -storepass $(/opt/netapp/essentials/bin/elytronstore.sh /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore jboss $(cat /opt/netapp/essentials/conf/server.properties | grep "keystore.token" | cut -c 16-)) -trustcacerts -file </path_to_the_cert_file/cert_name>.crt

  1.  Alternatively via the supported helper:

/opt/netapp/essentials/bin/cert.sh import </path_to_the_cert_file/cert_name>.crt

  1. Re-sync OCIE and restart AIQUM services so the new keystore is loaded by both JBoss and OCIE.
  2. Address the environmental root cause so the issue does not recur:
    • Confirm SELinux state with getenforce and, if it is Enforcing, review the AVC denials in /var/log/audit/audit.log against the AIQUM directories and add the required policy / labels.
    • Audit the Ansible / configuration-management playbooks that run against the AIQUM host and exclude the AIQUM-owned paths from generic chmod/chown/restorecon steps.

Partner Notes

partnerNotes_text

Additional Information

  • Files touched by the Generate CSR GUI flow :
    • /opt/netapp/essentials/bin/cert.sh
    • /opt/netapp/essentials/bin/certificatemanager/platform-base-display-key.jar
    • /opt/netapp/data/ocie/server.keystore
    • /opt/netapp/essentials/jboss/standalone/data/jbossCredStore.cs
    • /opt/netapp/essentials/bin/elytronstore.sh
    • /opt/netapp/essentials/conf/server.properties
    • /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore
  • The two server.keystore files (under /opt/netapp/data/ocie/ and /opt/netapp/essentials/jboss/server/onaro/cert/) must have an identical cksum.

Internal Notes

  • Source case: 2010649653 (UBS, AIQUM 9.16 on RHEL)
  • CPE escalation: CPE-12443 — AIQUM 9.16 — Generate CSR from GUI fails: BouncyCastle "cannot create signer: Key must not be null" (UBS, 2010649653)
  • Bug ID: CAIQUM-7005
  • Patch: a fixed keystoresetup.jar is available via mysupport — AIQUM 9.18D21
  • Both server.keystore copies must have the same cksum:

[root@scs001105021 ~]# cksum /opt/netapp/data/ocie/server.keystore
1751053047 9941 /opt/netapp/data/ocie/server.keystore
[root@scs001105021 ~]# cksum /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore
1751053047 9941 /opt/netapp/essentials/jboss/server/onaro/cert/server.keystore

  • On the source case the customer's Ansible / hardening modified file permissions on the AIQUM paths, and SELinux was in enforcing mode — both contributed
  • A temporary setenforce 0 during the remote session was used only to confirm the SELinux contribution and was reverted afterwards; long-term fix is to either exclude the AIQUM paths from the hardening playbook or to add the required SELinux policy.

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.