CHW-2478: AFF or FAS storage system fails to boot due to mSATA device failure

Issue

• The following storage systems experience controller disruption and fail to boot:
• AFF C190, AFF A150, AFF A200, AFF A220, AFF A300, AFF A700
• FAS2620, FAS2650, FAS2720, FAS2750, FAS2820, FAS8200, FAS9000
• Node goes down with one of the following:

PANIC: thread (xxx) on cpu hung for 4001 milliseconds in process xxx
Panic string: g_vfs_done(): rootfs.uzip read error - suspect boot device
Panic string: Process mgwd unresponsive for 620 seconds in process nodewatchdog
Could not load fat://boot0/X86_64/freebsd/image1/kernel:Device not found
ERROR: Error booting OS on: 'boot0' file: fat://boot0/X86_64/freebsd/image1/kernel (boot0,fat)
 

• Autoboot fails with the following error message:

BIOS Version: 11.7
{\{Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved. }}
Initializing System Memory ...
{\{Loading Device Drivers ... }}
{\{Configuring Devices ... }}
{\{CPU = 1 Processor(s) Detected. }}
{\{Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0) }}
{\{CPUID: 0x00050664. Cores per Processor = 12 }}
{\{32768 MB System RAM Installed. }}
{\{Boot Loader version 6.0.8 }}
{\{Copyright (C) 2000-2003 Broadcom Corporation. }}
{\{Portions Copyright (C) 2002-2018 NetApp, Inc. All Rights Reserved. }}
*ERROR: Failure to build endpoint environment variables. Autoboot will be disabled.*
 

• BMC event logs indicate the following errors:

[Controller.notice]: Appliance user command panic.
[IPMI.notice]: 001f | 02 | EVT: 6f406fff | Sensor 255 | Assertion Event, "Storage OS stop/shutdown" [BMC.critical]: Filer Reboots
[IPMI.notice]: (PUA) Enable power to all PCIe slots
[IPMI.notice]: (PUA) Enable power to all PCIe on board device
[IPMI.notice]: (PUA) P_stat :slots=0x0,onboard_devs=0x0,final
[IPMI.notice]: (PUA) Power status of all PCIe slots unchanged
[IPMI.notice]: 0023 | 02 | EVT: 6fc203ff | System_FW_Status | Assertion Event, "Memory Initialization done"
*[SysFW.warning]: Error loading ENV variables, creating defaults*
*[SysFW.warning]: Failed to read FW image on boot media: Device not found. Auto up*
[SysFW.warning]: date skipped
[IPMI.notice]: 0024 | 02 | EVT: 6fc220ff | System_FW_Status | Assertion Event, "Bootloader is running"
[BMC.critical]: Heartbeat stopped