Takeover is not possible: Mailbox disks are not healthy in GCP
Applies to
- Cloud Volumes ONTAP (CVO)
- Google Cloud Platform (GCP)
Issue
Errors on Node01:
Wed Jan 18 06:24:030000 [cluster-01: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]:Failover monitor: takeover of cluster-02 disabled (Mailbox disks are not healthy).
Wed Jan 18 06:24:050000 [cluster-01: fmmbx_instanceWorker: fmmb.io.policy.match:debug]:params: {'failureMediator': '0', 'success': '2', 'failurePartner': '0','failureLocal': '0', 'action': 'Read Ok', 'operation': 'Read', 'side':'Partner', 'failureUnknown': '0'} Wed Jan 18 06:24:050000 [cluster-01: fmmbx_instanceWorker:fmmb.redundancy.status:debug]: params: {'redundancyState': 'Partial', 'side':'Partner'} Wed Jan 18 06:25:010000 [cluster-01:monitor: monitor.globalStatus.critical:EMERGENCY]: Controller failover of cluster-02is not possible: Mailbox disks are not healthy.
Wed Jan 18 07:00:020000 [cluster-01: statd: cf.takeover.disabled:alert]: HA mode, buttakeover of partner is disabled due to reason : Mailbox disks are not healthy.
Errors on Node02:
Wed Jan 18 06:24:020000 [cluster-02: cf_main: cf.fsm.takeoverByPartnerDisabled:error]:Failover monitor: takeover of cluster-02 by cluster-01 disabled (Mailbox disksare not healthy).
Wed Jan 18 06:24:050000 [cluster-02: fmmbx_instanceWorker: fmmb.lock.disk.remove:info]:Disk 0f.30 removed from Partner mailbox set.
'uuid':'63766F75:63317577:63646576:2D766D31:2D646973:6B2D726F:6F740000:00000000:00000000:00000000','flag': '0x61', 'location': 'Partner node', 'node_attach_uuid': 'e0661a37-0766-11ed-8ff1-85e956ba8507','disk': '0d.18', 'side': 'Partner'} Wed Jan 18 06:24:050000 [cluster-02: fmmbx_instanceWorker:fmmb.online.disk:debug]: params: {'nodeAttachId': '0', 'uuid':'63766F75:63317577:63646576:2D766D31:2D646973:6B2D726F:6F74636F:70790000:00000000:00000000','flag': '0x61', 'location': 'Local node', 'node_attach_uuid':'ddf19d54-0766-11ed-b7f8-cbc03aeb40cd', 'disk': '0b.11', 'side': 'Partner'} WedJan 18 06:24:080000 [cluster-02:fmmbx_instanceWorker: fmmb_disk_io_replyFailed_1:debug]: params: {'status':'disk does not exist', 'handle': '0x0',
Wed Jan 18 06:24:230000 [cluster-02: fmmbx_instanceWorker: fmmb.online.disk:debug]: params:{'nodeAttachId': '0', 'uuid':'34363435:36316534:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000','flag': '0x21', 'location': 'Mediator node', 'node_attach_uuid':'00000000-0000-0000-0000-000000000000', 'disk': '0f.5', 'side': 'Partner'} WedJan 18 06:24:260000 [cluster-02:fmmbx_diskOpsManager: fmmb_disk_io_replyFailed_1:debug]: params: {'status': 'novalid path to disk', 'handle': '0x0', 'uuid':'34363435:36316534:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000','io_state': '1', 'flag': '0x14', 'operation': 'MBX_DISKIO_READ_1A', 'side':'Partner', 'disk_name': '0f.5'} Wed Jan 18 06:24:300000 [cluster-02: fmmbx_instanceWorker:fmmb.lock.disk.remove:info]: Disk 0f.5 removed from Partner mailbox set.
Wed Jan 18 06:25:020000 [cluster-02: monitor: monitor.globalStatus.critical:EMERGENCY]:Controller failover of cluster-01 is not possible: Mailbox disks are nothealthy.