No access to HTTPS services on new cluster nodes, because SSL certificate did not replicate

Applies to

Issue

• Curl error:

  • Closing connection 0
    curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to x.x.x.x:443

• Varonis errors:

  • 2025-04-15 14:40:36.7100000ﾠDTWCP001GEN127ﾠVaronis.SystemDiscovery.WebServiceﾠ4336ﾠ3ﾠNetapp::NetApp::{ctor}::<lambda_b9d620ee355154c945548d16c6f1bea7>::operator (): Filer 0. Can't connect to the filer (host: svms3test) using ONTAPI (HRESULT: 1222).
    Failed to invoke command on server: svms3test.
    Command:
    <system-get-version/>

  • Error: 1(0x00000001) Description: No permission to use 'hostsequiv' authentication, must be root.

• ASUP APACHE-ERROR.GZ error on the symptomatic nodes:

  • [Wed Apr 23 08:00:08.012309 2025 +0000] [ssl:error] [pid 9250:tid 34401860864] [client x.x.x.x:62447] [vserver 32] Failed to initialize SSL context
    [Wed Apr 23 08:00:08.013457 2025 +0000] [ssl:notice] [pid 9250:tid 34401860864] [client x.x.x.x:62448] [vserver 32] No server certificate chain is configured for this vserver
    [Wed Apr 23 08:00:08.013481 2025 +0000] [ssl:notice] [pid 9250:tid 34401860864] [client x.x.x.x:62448] [vserver 32] Certificate-based client authentication is not configured for this vserver
    [Wed Apr 23 08:00:08.014163 2025 +0000] [ssl:emerg] [pid 9250:tid 34401860864] AH02562: Failed to configure certificate 127.0.0.1:0 (with chain), check /mroot/etc/cluster_config/vserver/.vserver_32/config/etc/certificates/ssl/server/150+17BE379C1811D356+svms3test/server.crt
    [Wed Apr 23 08:00:08.014175 2025 +0000] [ssl:emerg] [pid 9250:tid 34401860864] SSL Library Error: error:80000002:system library::No such file or directory (calling fopen(/mroot/etc/cluster_config/vserver/.vserver_32/config/etc/certificates/ssl/server/150+162618A776ACDAF8+svmncmain2/server.crt, r))

  • Vserver ID shown in the APACHE-ERROR.GZ error matches the vserver that HTTPS clients cannot connect to

    • nas-cm913::> vserver show -id 32
                                     Admin      Operational Root
      Vserver     Type    Subtype    State      State       Volume     Aggregate
      ----------- ------- ---------- ---------- ----------- ---------- ----------
      svms3test   data    default    running    running     svms3test_ n2_aggr1
                                                            root

  • Folder name shown in the APACHE-ERROR.GZ error matches the serial number of the certificate that is assigned to the vserver's SSL configuration

    • nas-cm913::> ssl show -vserver svms3test   
        (security ssl show)
                                               Vserver: svms3test   
                         Server Certificate Issuing CA: svms3test   
                      Server Certificate Serial Number: 17BE379C1811D356
                        Server Certificate Common Name: svms3test   
                     SSL Server Authentication Enabled: true
                     SSL Client Authentication Enabled: false
      Online Certificate Status Protocol Validation Enabled: false
      URI of the Default Responder for OCSP Validation:
      Force the Use of the Default Responder URI for OCSP Validation: false
                              Timeout for OCSP Queries: 10s
       Maximum Allowable Age for OCSP Responses (secs): unlimited
      Maximum Allowable Time Skew for OCSP Response Validation: 5m
                       Use a NONCE within OCSP Queries: true
      

• Packet trace shows that ONTAP does not participate in TLS handshake

  • 108436 2025-04-15 14:40:35.399239 x.x.x.x y.y.y.y TCP 66 633 49823 → 443 [SYN, ECE, CWR] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM
    108437 2025-04-15 14:40:35.399508 y.y.y.y x.x.x.x TCP 66 633 443 → 49823 [SYN, ACK, ECE] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM
    108438 2025-04-15 14:40:35.399556 x.x.x.x y.y.y.y TCP 54 633 49823 → 443 [ACK] Seq=1 Ack=1 Win=2097920 Len=0
    108439 2025-04-15 14:40:35.400087 x.x.x.x y.y.y.y TLSv1 356 633 Client Hello
    108440 2025-04-15 14:40:35.400804 y.y.y.y x.x.x.x TCP 60 633 443 → 49823 [FIN, ACK] Seq=1 Ack=303 Win=65792 Len=0
    108441 2025-04-15 14:40:35.400833 x.x.x.x y.y.y.y TCP 54 633 49823 → 443 [ACK] Seq=303 Ack=2 Win=2097920 Len=0
    108442 2025-04-15 14:40:35.400865 x.x.x.x y.y.y.y TLSv1 61 633 Alert (Level: Fatal, Description: Decode Error)
    108443 2025-04-15 14:40:35.400928 x.x.x.x y.y.y.y TCP 54 633 49823 → 443 [FIN, ACK] Seq=310 Ack=2 Win=2097920 Len=0
    108444 2025-04-15 14:40:35.401167 y.y.y.y x.x.x.x TCP 60 633 443 → 49823 [RST] Seq=2 Win=0 Len=0
    108445 2025-04-15 14:40:35.401217 y.y.y.y x.x.x.x TCP 60 633 443 → 49823 [RST] Seq=2 Win=0 Len=0