CHW-2478: AFF or FAS storage system fails to boot due to mSATA device failure
Issue
- The following storage systems experience controller disruption and fail to boot:
- AFF C190, AFF A150, AFF A200, AFF A220, AFF A300, AFF A700
- FAS2620, FAS2650, FAS2720, FAS2750, FAS2820, FAS8200, FAS9000
- Node goes down with one of the following:
PANIC: thread (xxx) on cpu hung for 4001 milliseconds in process xxx Panic string: g_vfs_done(): rootfs.uzip read error - suspect boot devicePanic string: Process mgwd unresponsive for 620 seconds in process nodewatchdogCould not load fat://boot0/X86_64/freebsd/image1/kernel:Device not foundERROR: Error booting OS on: 'boot0' file: fat://boot0/X86_64/freebsd/image1/kernel (boot0,fat)- Autoboot fails with the following error message:
BIOS Version: 11.7Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved. Initializing System Memory ...Loading Device Drivers ... Configuring Devices ... CPU = 1 Processor(s) Detected. Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0) CPUID: 0x00050664. Cores per Processor = 12 32768 MB System RAM Installed. Boot Loader version 6.0.8 Copyright (C) 2000-2003 Broadcom Corporation. Portions Copyright (C) 2002-2018 NetApp, Inc. All Rights Reserved. *ERROR: Failure to build endpoint environment variables. Autoboot will be disabled.*- BMC event logs indicate the following errors:
[Controller.notice]: Appliance user command panic.[IPMI.notice]: 001f | 02 | EVT: 6f406fff | Sensor 255 | Assertion Event, "Storage OS stop/shutdown" [BMC.critical]: Filer Reboots [IPMI.notice]: (PUA) Enable power to all PCIe slots [IPMI.notice]: (PUA) Enable power to all PCIe on board device [IPMI.notice]: (PUA) P_stat :slots=0x0,onboard_devs=0x0,final [IPMI.notice]: (PUA) Power status of all PCIe slots unchanged [IPMI.notice]: 0023 | 02 | EVT: 6fc203ff | System_FW_Status | Assertion Event, "Memory Initialization done" *[SysFW.warning]: Error loading ENV variables, creating defaults**[SysFW.warning]: Failed to read FW image on boot media: Device not found. Auto up*[SysFW.warning]: date skipped [IPMI.notice]: 0024 | 02 | EVT: 6fc220ff | System_FW_Status | Assertion Event, "Bootloader is running"[BMC.critical]: Heartbeat stopped