Azure maintenance causes node takeover and alerts on CVO

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 177

Visibility:: Public

Votes:: 0

Category:: cloud-volumes-ontap-cvo

Specialty:: microsoft_azure

Last Updated:

Applies to

BlueXP
Cloud Volumes ONTAP (CVO) Azure

Issue

Multiple alerts are detected on Node-02 by the specific customer monitoring tool

First Arrival: 2025-05-24 17:07:16 JST

Last Arrival: -

Customer: xxxx

Hostname: Node-02

Message:

CONFIDENTIALITY=NetApp Confidential

GENERATED_ON=Sat May 24 17:07:04 +0900 2025

VERSION=NetApp Release 9.11.1P12: Fri Sep 22 07:58:50 EDT 2023

SYSTEM_ID=2319499287

SERIAL_NUM=9082014000000000xxx

HOSTNAME=Node-02

SEQUENCE=3756

PARTNER_SYSTEM_ID=xxxxxx

PARTNER_HOSTNAME=Node-01

BOOT_CLUSTERED='true'

CVO health check shows the below message on System Manager:

Warning: Node "Node-02" is waiting for a giveback from Node "Node-01".

clam.node.ooq: Node (name=Node-01, ID=1000) is out of "CLAM quorum" (reason=seen by HA partner).

Failed to initiate giveback. Run the "storage failover show-giveback" command for more

monitor.globalStatus.critical: Controller failover of Node is not possible: partner booting.

Node-02 was taken over by Node-01 and subsequently rebooted.

24May2025 17:06:36 sfo_takenOver_relocAllReq partner_name="Node-02" partner_sysid="xxxxx" 24May2025 17:06:36 ha_takenOver_stateChng old_state="NOT_IN_TAKEOVER" new_state="SFO_PHASE_INPROG" 24May2025 17:06:36 clam_peer_halting hostnode="Node-02" nodeid="xxxx"

BlueXP shows the below alert:

Degraded - Show Details HA cluster is not highly available.The node giveback was not completed.

SSL/TLS communication check failed.

[node: ktlsd: ktls.failed:notice]: "The TLS connections have failed several times with remote host 'xx.xx.xx.xx' in IPspace 'xxxx', for which the latest reason given is: OpenSSL: error:0A000086:SSL routines::certificate verify failed."

Cause

Azure maintenance cause node takeover and reboot, resulting in related alerts.

Note: Before Azure maintenance, the following AutoSupport is sent:

::> system node autosupport history show -fields subject,generated-on

node    seq-num destination subject                                                                                                          generated-on

------- ------- ----------- ---------------------------------------------------------------------------------------------------------------- ------------------

...

Node-01 11      smtp        USER_TRIGGERED (TEST:MAINT=1h Cloud Provider Maintenance Event. Status: scheduled. Type: freeze. VM: <VM-NAME>.) 5/24/2025 17:07:04

Node-01 11      http        USER_TRIGGERED (TEST:MAINT=1h Cloud Provider Maintenance Event. Status: scheduled. Type: freeze. VM: <VM-NAME>.) 5/24/2025 17:07:04

Node-01 11      noteto      USER_TRIGGERED (TEST:MAINT=1h Cloud Provider Maintenance Event. Status: scheduled. Type: freeze. VM: <VM-NAME>.) 5/24/2025 17:07:04

Solution

Perform a manual giveback operation to resolve the issue.

Partner Notes

partnerNotes_text

Additional Information

Sometimes the auto giveback be completed without issue , in such case no further action is required.

Internal Notes

N/A