Cloud Tier not available after StorageGRID certificate expired in ONTAP
Applies to
- FabricPool
- ONTAP
- StorageGRID
Issue
- The object-store becomes unavailable after StorageGRID certificate expired in ONTAP
- Enabling the certificate validation is failing with error:
Cannot verify availability of the object store from node NodeAa. Reason: Cannot verify the certificate given by the object store server.
It is possible that the certificate has not been installed on the cluster.
Use the 'security certificate install -type server-ca' command to install it..
EMS.log:
[NodeAa: ktlsd: ktls.failed:notice]: "The TLS connections have failed several times with remote host '10.3.3.123' in IPspace '1234', for which the latest reason given is: OpenSSL: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed."
[nodeAa: OscHighPriThreadPool: object.store.unavailable:EMERGENCY]: Unable to connect to the object store "<Fabric Pool>" from node <node ID>. Reason: Connection unavailable.
SKtrace.log:
2021-08-19T16:09:07Z 12420711494086901 [12:0] OSC_ERR: logFailedCmd:356 HEAD Cmd ID:128 failed with OSC error: -336134278
2021-08-19T16:09:07Z 12420711494089813 [12:0] OSC_ERR: logFailedCmd:367 Stats of HEAD Cmd ID:128, wait 125483us, start-send 126651us, 1stbyte 0us, 1st-lastbyte 0us
2021-08-19T16:09:08Z 12420713417698175 [4:0] OSC_ERR: logFailedCmd:351 PUT Cmd ID:128 name:config_test_b01af070-8ea1-11e8-9eb6-529269fb1459 failed with OSC error: -336134278
2021-08-19T16:09:08Z 12420713417701232 [4:0] OSC_ERR: logFailedCmd:367 Stats of PUT Cmd ID:128, wait 127049us, start-send 127167us, 1stbyte 0us, 1st-lastbyte 0us
2021-08-19T16:09:09Z 12420715329109906 [15:0] OSC_ERR: logFailedCmd:351 PUT Cmd ID:128 name:config_test_b01af070-8ea1-11e8-9eb6-529269fb1459 failed with OSC error: -336134278
2021-08-19T16:09:09Z 12420715329112830 [15:0] OSC_ERR: logFailedCmd:367 Stats of PUT Cmd ID:128, wait 124031us, start-send 124137us, 1stbyte 0us, 1st-lastbyte 0us
2021-08-19T16:09:10Z 12420717245482467 [13:0] OSC_ERR: logFailedCmd:351 PUT Cmd ID:128 name:config_test_b01af070-8ea1-11e8-9eb6-529269fb1459 failed with OSC error: -336134278
2021-08-19T16:09:10Z 12420717245485368 [13:0] OSC_ERR: logFailedCmd:367 Stats of PUT Cmd ID:128, wait 123929us, start-send 124031us, 1stbyte 0us, 1st-lastbyte 0us
2021-08-19T16:09:12Z 12420719162024136 [13:0] OSC_ERR: logFailedCmd:351 PUT Cmd ID:128 name:config_test_b01af070-8ea1-11e8-9eb6-529269fb1459 failed with OSC error: -336134278
2021-08-19T16:09:12Z 12420719162027261 [13:0] OSC_ERR: logFailedCmd:367 Stats of PUT Cmd ID:128, wait 123943us, start-send 124047us, 1stbyte 0us, 1st-lastbyte 0us