Multi disk failure PANIC during partner node's parts replacement

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 174

Visibility:: Public

Votes:: 0

Category:: fas-systems

Specialty:: hw

Last Updated:

Applies to

FAS27x0
ONTAP 9
Advanced Disk Partitioning (ADP)

Issue

During partner node parts replacement, remove PCM from chassis, at that time partner takeover node occurred Multi Disk Panic.

EMS log:

PANIC: aggr pro_aggr01: raid volfsm, fatal multi-disk error. raid type raid_dp Group name plex0/rg0 state NORMAL 3 disks failed in the group. Disk 0b.00.0P1 Shelf 0 Bay 0 [NETAPP X343_STBTE1T8A10 NA02] S/N [XXXXXXXXXX001] UID [6000C500:XXXXXXXX:500A0981:00000001:00000000:00000000:00000000:00000000:00000000:00000000] error disk does not exist. Disk 0b.00.1P1 Shelf 0 Bay 1 [NETAPP X343_STBTE1T8A10 NA02] S/N [XXXXXXXXXX001] UID [6000C500:XXXXXXXX:500A0981:00000001:00000000:00000000:00000000:00000000:00000000:00000000] error disk does not exist. Disk 0b.00.3P1 Shelf 0 Bay 3 [NETAPP X343_STBTE1T8A10 NA02] S/N [XXXXXXXXXX001] UID [6000C500:XXXXXXXX:500A0981:00000001:00000000:00000000:00000000:00000000:00000000:00000000] error disk does not exist. in SK process config_thread on release 9.4P4 (C) on Tue Jul 12 16:43:52 JST 2022 version: 9.4P4: Thu Nov 1 11:20:54 EDT 2018

After Panic, a node is booting up but is missing 3 disks and aggregate is failed/offline status.

sysconfig -r:

Aggregate pro_aggr01 (failed, raid_dp, partial) (block checksums) Plex /pro_aggr01/plex0 (offline, failed, inactive) RAID group /pro_aggr01/plex0/rg0 (partial, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity FAILED N/A 1689530/ - parity FAILED N/A 1689530/ - data 0b.00.2P1 0b 0 2 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 data FAILED N/A 1689530/ - data 0b.00.4P1 0b 0 4 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 data 0b.00.5P1 0b 0 5 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 data 0b.00.11P1 0b 0 11 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 data 0b.00.7P1 0b 0 7 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 data 0b.00.8P1 0b 0 8 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 data 0b.00.9P1 0b 0 9 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 data 0b.00.10P1 0b 0 10 SA:B 0 SAS 10000 1689530/3460157952 1689538/3460174336 Raid group is missing 3 disks.