System disruption occurs after temporarily unable to read or write to boot device on multiple AFF and FAS platforms
Applies to
- AFF A200 | FAS2620 | FAS2650
- AFF A220 | AFF C190 | FAS2720 | FAS2750
- AFF A300 | FAS8200
- FAS8300
- AFF A700 | FAS9000
- AFF A900 | FAS9500
- mSATA boot device (P/N X3320A)
- Fails to Boot ONTAP OS
Issue
The node with the boot device experiencing the issue will panic or fail to boot ONTAP.
Panic Strings
PANIC: thread (xxx) on cpu hung for 4001 milliseconds in process xxx
PANIC: thread (swi4: clock) on cpu 0 hung for 4001 milliseconds in process swi4: clock on release 9.5P1 (C) on Thu Jun 11 03:41:13 EDT 2020PANIC: thread (md0.uzip) on cpu 0 hung for 4001 milliseconds in process md0.uzip on release 9.3P20 (C) on Tue Feb 2 22:28:58 JST 2021Panic string: Process mgwd unresponsive for 620 seconds in process
PANIC: g_vfs_done(): rootfs.uzip read error - suspect boot device in process g_up on release 9.7P4 (C) on Tue Aug 11 02:55:33 EDT 2020
EMS Logs
You may also see the following prior to the panic in ems log:
cam: cam.timeout.retry:notice]: CAM device driver I/O timeout. Details: CAM command timeout (retrying), Device ada0 (4 retries left). Device ID: SPG2153013S. Command: WRITE_DMA. ACB: ca 00 bb 88 88 40 00 00 00 00 70 00.
Console Startup Messages During Failure to Boot
Following are console message that could be seen during an attempted ONTAP boot.
- Node fails to boot with console log boot device errors:
g_vfs_done():ada0s1[READ(offset=1754742784, length=8192)]error = 5
ada0: CAM_CMD_TIMEOUT, retrying (4 retries left)
ada0: CAM_CMD_TIMEOUT, retrying (3 retries left)
ada0: CAM_CMD_TIMEOUT, retrying (2 retries left)
ada0: CAM_CMD_TIMEOUT, retrying (1 retry left)
Unable to read directoryRetry #5 of 5: /sbin/fsck_msdosfs /dev/ada0s1
bootarg.init.verify_checksum="true"
** /dev/ada0s1
** Phase 1 - Read and Compare FATs
** Phase 2 - Check Cluster Chains
** Phase 3 - Checking Directories
Unable to read directory
********************************************************
* ALERT: Boot device file system corruption detected: *
* FAT filesystem check exited with status 8 *
* *
* File system consistency check failed. *
* The node will be halted. *
* Contact Support for assistance. *
********************************************************
Uptime: 17s
System halting...
- Failure to reboot examples:
Could not load fat://boot0/X86_64/freebsd/image1/kernel:Device not found ERROR: Error booting OS on: 'boot0' file: fat://boot0/X86_64/freebsd/image1 /kernel (boot0,fat) Could not load fat://boot0/X86_64/freebsd/image1/kernel:Device is open ERROR: Error booting OS on: 'boot0' file: fat://boot0/X86_64/freebsd/image1/kernel (boot0,fat)
ERROR: Failure to build endpoint environment variables. Autoboot will be disabled.
- If the platform is a nosavecore platform, it will not be possible to save the core