System repeatedly reboots with mailbox disks error

Applies to

• AFF-A400
• ONTAP 9.9.1P2

Issue

• System boots up to Waiting for giveback with Reservation conflict error after a power cycle.

Reservation conflict found on this node's disks!
 Local System ID: 123456789
 Press Ctrl-C for Maintenance menu to release disks.
 [node2:cf.fmns.skipped.disk:notice]: While releasing the reservations in "Waiting For Giveback" state Failover Monitor Node State(fmns) module skipped the disk 0a.00.8 that is owned by 123456789 and reserved by 987654321.

 [node2:cf.disk.ResvFail:ALERT]: Disk 0d.00.9P3 has been reserved by the High Availability (HA) partner as part of a takeover operation.
 [node2:cf.disk.ResvTakeOver:notice]: This node will wait for giveback and the disk reservations to be released.
 Disk reservations have been released
 Waiting for giveback...(Press Ctrl-C to abort wait)
 Waiting for giveback...(Press Ctrl-C to abort wait)
 Waiting for giveback...(Press Ctrl-C to abort wait)

• After rebooting the node from Waiting for giveback status, system does not boot up and reboots repeatedly with All Local mailbox disks are inaccessible error.

 Rebooting ...
 Uptime: 6m48s
 BIOS Version: 16.4
 PEI start.
 CPU PEI initialization.
 Wait BMC self-test result.
 BMC self-test: OK.
 UPI initialization.
 CPU initialization.
 Running quick memory initialization.
 SPI FLASH: Primary BIOS
 PEI end.
 DXE start.
 USB initialization.
 PCI host bridge initialization.
 CSM initialization.
 PCI Bus initialization start.
 BDS start.
 Console output devices connect.

 Ready to boot.
 
 Boot Loader version 6.4.9
 Copyright (C) 2000-2003 Broadcom Corporation.
 Portions Copyright (C) 2002-2021 NetApp, Inc. All Rights Reserved.
 
 ACPI RSDP Found at 0x6cc29000
 
 
 Starting AUTOBOOT press Ctrl-C to abort...
 Loading X86_64/freebsd/image2/kernel:0x200000/1147616 0x319000/10792232 0xf64000/3948368 0x1327f50/4387448 0x200240/1016 Entry at 0xffffffff80319000
 Loading X86_64/freebsd/image2/platform.ko:0x1758000/4068960 0x1b39660/628568
 Starting program at 0xffffffff80319000
 ---<<BOOT>>---
 NetApp Data ONTAP 9.9.1P2
 IPMI device unit 0 rev. 1, firmware rev. 13.04, version 2.0, device support mask 0xbf
 IPMI device unit 1 rev. 1, firmware rev. 13.04, version 2.0, device support mask 0xbf
 Copyright (C) 1992-2021 NetApp.
 All rights reserved.
 *******************************
 *                             *
 * Press Ctrl-C for Boot Menu. *
 *                             *
 *******************************

 [node2:fmmb.disk.notAccsble:notice]: All Local mailbox disks are inaccessible.

 [node2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of node1 disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).
 [node2:kern.syslog.msg:notice]: domain xing mode: off, domain xing interrupt: false
 [node2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of node1 disabled (unsynchronized log).

• HA interconnect ports are up on partner node during the impaired node's reboot.

SYSCONFIG-A

    slot 0: 10G/25G Ethernet Controller CX5
        e0a MAC Address:    xx:xx:xx:xx:xx:xx (auto-25g_cr-fd-up)
            SFP Vendor:         Amphenol
            SFP Part Number:    NDCCGF-N103
            SFP Serial Number:  APF1234567
        e0b MAC Address:    xx:xx:xx:xx:xx:xx (auto-25g_cr-fd-up)
            SFP Vendor:         Amphenol
            SFP Part Number:    NDCCGF-N103
            SFP Serial Number:  APF7654321