Node shuts down and fails to boot due to 'Multiple fans failed'

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 912

Visibility:: Public

Votes:: 0

Category:: aff-series

Specialty:: HW

Last Updated:

Applies to

Baseboard Management Controller firmware lower than 11.5
- AFF C190, AFF A220, FAS2720, FAS2750
Service Processor firmware lower than 5.8
- AFF A300, AFF A200, FAS8200, FAS2650, FAS2620

Issue

Both nodes shut down and failed to boot
The node shuts down with the following message:

Sat May 08 22:20:24 +0100 [node-1: env_mgr: monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Environmental Reason Shutdown (Multiple fans failed) Sat May 08 22:20:27 +0100 [node-1: mgwd: mgwd.notify.halt.result:info]: MGWD able to notify CLAM on its HA partner node that this node is undergoing a planned shutdown (reason: E). Error: - Sat May 08 22:20:34 +0100 [node-1: env_mgr: monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Status of fans is unknown for 90 seconds. Shutting down now.

AutoSupports may trigger alerts:

HA Group Notification (CONTROLLER TAKEOVER COMPLETE HALT) NOTICE

HA Group Notification (Health Monitor process cphm: CriticalFruMultiFaultAlert[PSQ094195000111]) ALERT

If nodes fail to boot and controller reseat is attempted, nodes may remain down/unable to boot
May see the following on boot in console logs:

Initializing System Memory ... Loading Device Drivers ... Configuring Devices ... Waiting for SP ...

IPMI:Read midplane FRU 0 product info:timeout IPMI:Read midplane FRU 0 product info:failed Waiting for SP ...

IPMI:Get midplane FRU 1 inventory:timeout SP failure. Resetting SP from primary FW. This can take a few minutes Waiting for SP ... SP failure. Resetting SP from backup FW. This can take a few minutes Waiting for SP ... Failed to recover SP IPMI:Get midplane FRU 1 inventory:failed

IPMI:Get controller FRU inventory:failed IPMI:Get midplane FRU 0 inventory:failed Configuring Devices ... IPMI PCI Slot Control failed.

If boot succeeds, node may complain on different sensors and shut down again:

Mon May 24 10:07:52 GMT [nvram.hw.initWarn:WARNING]: NVRAM hardware initialization: Failed to get Battery FRU info.

May 24 10:11:19 [node-1:sp.ipmi.lost.shutdown:EMERGENCY]: SP heartbeat stopped and cannot be recovered. To prevent hardware damage and data loss, the system will shut down in 2 minutes. May 24 10:13:19 [node-1:monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Environmental Reason Shutdown (System reboot to recover the SP)