AFF/FAS80x0 - Chassis fan failure alerts reported even after fan replacement
Applies to
- FAS/AFF 80x0
- AFF A800
- Data ONTAP 8
- ONTAP 9
- SysFan or IOFan Failed
Issue
- Multiple system fans failed.
- Node Environmental Shutdown
EMS.logshows:
[node-n1: env_mgr: callhome.c.fan.fru.fault:error]: Call home for CHASSIS FAN FRU FAILED: SysFan3 F2
[node-n1: env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F2 (failed)
[node-n1: env_mgr: monitor.chassisFanFail.xMinShutdown:EMERGENCY]: Multiple Chassis Fan failure: System will shut down in 2 minutes.
[node-n1:monitor.globalStatus.critical:EMERGENCY]: Multiple fans has failed: Sysfan3 F2, Sysfan3 F1.
[node-n1:monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Environmental Reason Shutdown (Multiple fans failed)
- This platform has 3 SysFans for each controller (6 in total)
- If there is an IOXM installed, there will be 3 SysFans and 3 IOFans
- SysFans are in front of the contoller
- IOFans are in front of the IOXM
- After swapping the SysFan with the error with a known working SysFan from another bay, the issue remains on the same bay.
- If the issue is just with one SysFan, the other node should be up and running all the time. Do not use the partner node SysFans for testing.
- Messages like the following can be found at
SYSCONFIG -M:
!Chassis!021530002902!111-01459!E0!
!FAN1!021528004801!441-00037!D0!
!FAN2!021528004775!441-00037!D0!
!FAN3: Error reading FRU EEPROM
!FAN4!021528005758!441-00037!D0!
!FAN5!021528004271!441-00037!D0!
!FAN6!021528004776!441-00037!D0!- In the
SP events:
Aug 12 00:02:08 (none) seld[550]: FPGA event 21 : Fan Reset 3 asserted
Aug 12 00:02:09 (none) seld[550]: FPGA event 21 : Fan Reset 3 de-asserted
Aug 12 00:02:15 (none) : [353 INFO]FanDaemon: Fan module 3 was reset (waiting some time before accessing)
Aug 12 00:02:20 (none) : [353 INFO]FanDaemon: Fan module 3 reset should be complete (will attempt access next cycle)
Aug 12 00:02:26 (none) : [353 WARNING][Porting/platform/PDKFan.c:1661]FanDaemon: Failure to read FanModuleData from fan module 3
Aug 12 00:02:26 (none) : [353 WARNING][Porting/platform/PDKFan.c:1676]FanDaemon: Reset Fan Module 3
Aug 12 00:02:26 (none) : [353 INFO]Fan Module 3 Reset- Errors remain on the same SysFan bay after swapping with a known good fan.