Node shuts down and fails to boot due to 'SP IPMI failure'

Applies to

• AFF A700 / FAS9000
• Service Processor firmware version below 4.9

Issue

• Both nodes shut down and fail to boot
• Motherboards re-seated but nodes are held at LOADER prompt
• The node shuts down with the following message:

sp.ipmi.lost.shutdown:EMERGENCY]: SP heartbeat stopped and cannot be recovered. To prevent hardware damage and data loss, the system will shut down in 2 minutes.

• An Autosupport maybe seen with

HA Group Notification (SP HBT STOPPED) ALERT

• Upon attempt to boot:

Initializing System Memory ...
Loading Device Drivers ...
Waiting for SP ...
IPMI:Enable PCI slots:timeout
SP failure. Resetting SP from primary FW. This can take a few minutes
Waiting for SP ...
SP recovered successfully after a reset from primary FW image
Waiting for SP ...

IPMI:Enable PCI slots:timeout
SP failure. Resetting SP from backup FW. This can take a few minutes
Waiting for SP ...
SP recovered successfully after a reset from backup FW image
Waiting for SP ...

IPMI:Enable PCI slots:timeout
Failed to recover SP
IPMI PCI Slot Control failed.

IPMI PCI Slot Configuration failed.

Configuring Devices ...
IPMI:Get controller FRU inventory:failed
IPMI:Get midplane FRU 0 inventory:failed
IPMI:Get NVRAM FRU inventory:failed

CPU = 2 Processor(s) Detected.
  Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (CPU 0)
  CPUID: 0x000406F1. Cores per Processor = 18
  Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (CPU 1)
  CPUID: 0x000406F1. Cores per Processor = 18
524288 MB System RAM Installed.
SATA (AHCI) Device: SV9MST6D120GLM41NP

Boot Loader version 6.0.12
Copyright (C) 2000-2003 Broadcom Corporation.
Portions Copyright (C) 2002-2021 NetApp, Inc. All Rights Reserved.

BIOS POST Failure(s) detected: SP IPMI failure. Abort AUTOBOOT