AllPathToOneEndOfStack_Alert appears after force takeover and giveback

Applies to

• FAS8200
• ONTAP 9
• Service Processor (SP)

Issue

• Regarding scheduled shutdown, system halt using the no-takeover option.

EMS log:

Sun Apr 16 08:01:09 JST [node-02: svc_queue_thread: clam.notify.halt.result:debug]: CLAM was able to notify its HA partner node that the local node is undergoing a planned shutdown (reason: Local node shutting down). Error: 0
Sun Apr 16 08:01:55 JST [node-02: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of node-01 disabled (partner halted in notakeover mode).
Sun Apr 16 08:07:39 JST [node-02: vifmgr: callhome.clus.net.degraded:alert]: Call home for CLUSTER NETWORK DEGRADED: Cluster LIF Not Assigned to Any Port - Cluster LIF node-02_clus1 (node node-02) is not assigned to any port.

• After booting up, system node-01 cannot boot up, so trying to power cycle the system using SP, but it cannot boot up due to BIOS POST Failure.

SP console output:

SP node-01> system power cycle
This command will trigger a destage
of ONTAP and will take some time to complete after which the system will restart. Use "system reset" or "system reset current" to
avoid the delay. Continue? [y/n] y

SP node-01> system console
Type Ctrl-D to exit.
Loading Device Drivers ...
Configuring
Devices ...

CPU = 1 Processor(s) Detected.
  Intel(R) Xeon(R) CPU D-1587 @ 1.70GHz (CPU 0)
  CPUID: 0x00050664. Cores per
Processor = 16
65536 MB System RAM Installed.
SATA (AHCI) Device: SV9MST6D120GLM41NP

Boot Loader version 6.0 
Copyright (C)
2000-2003 Broadcom Corporation.
Portions Copyright (C) 2002-2016 NetApp, Inc. All Rights Reserved.

BIOS POST Failure(s) detected:
PCIe device missing error detected. Abort AUTOBOOT
LOADER-A>

• Node-01 services are in a down state, so force takeover from node-01.

::> set -privilege advanced
::*> storage failover takeover -option force -ofnode node-01 -skip-lif-migration-before-takeover true

• After takeover, services are recovered and node-01 is in a Waiting for Giveback status.
• Perform giveback from node-02, but after that, slot status is misconfigured and the AllPathToOneEndOfStack_Alert appears.

Node node-02
Monitor node-connect
Alert ID AllPathToOneEndOfStack_Alert
Alerting Resource 1
Subsystem SAS-connect
Indication Time Mon May 01 09:25:02 2023
Perceived Severity Major
Probable Cause Cable_tamper
Description Controller node-02 is connected to only one end of stack 1 through disk shelf 1.20.
Corrective Actions 1. Consult the guide applicable to your IOM6 disk shelf to review cabling rules and complete the SAS cabling worksheet for your system.
2. Connect controller node-02 to the first and last disk shelves of stack 1.
3. Verify that controller node-02 is cabled to IOM A and IOM B of stack 1.
4. Contact technical support if the alert persists.

Possible Effect A single disk shelf failure within stack 1 might cause controller node-02 to lose access to multiple shelves in the stack.

• "sysconfig -ac" before the occurrence of the issue:

sysconfig: slot 1 OK: X2071A: 4x12Gb miniSAS HD HBA (PM8072)
sysconfig: slot 2 OK: X1049C: PCI-E Quad 10/100/1000 Ethernet 82580(v3.29 and above)
sysconfig: slot 3 OK: X3311A: Samsung M.2 1TB NVMe Drive
sysconfig: No shelf configuration errors detected.
sysconfig: There are no configuration errors.

• "sysconfig -ac" before the occurrence of the issue:

sysconfig: Samsung M.2 1TB NVMe Drive card (PN X3311A) in slot 1 must be in one of these slots: 3,4.
sysconfig: No shelf configuration errors detected.