Two disks partitioned simultaneously and displaying "SpareLow" during disk failure
Applies to
- FAS2620
- 9.9.1P15
Issue
- During a disk failure, two disks are partitioned simultaneously, resulting in the "SpareLow" condition.
- disk failed.
Mon Apr 01 14:46:34 +0900 [node-01: config_thread: callhome.fdsk.fault:error]: Call home for FILESYSTEM DISK FAILED Shelf 10, Bay 9, Model [X380_HLBRE10TA07], S/N [XXXXXXXX]
- Disk partition on the first spare disk.
[?] Mon Apr 01 14:46:35 +0900 [node-01: config_thread: raid.partition.disk:notice]: Disk partition successful on Disk 0a.10.2 Shelf 10 Bay 2 [NETAPP X380_STATE10TA07 NA00] S/N [XXXXXXXX] UID [5000C500:A75AC483:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [1]=2372885888.
[?] Mon Apr 01 14:46:36 +0900 [node-01: disk_layout_admin: disk.layout.changed:debug]: Detected partition layout change on disk 0a.10.2.
[?] Mon Apr 01 14:46:36 +0900 [node-01: dmgr_thread: raid.disk.inserted:info]: Disk 0a.10.2 Shelf 10 Bay 2 [NETAPP X380_STATE10TA07 NA00] S/N [XXXXXXXX] UID [5000C500:A75AC483:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] has been inserted into the system
- After partitioning spare disk, this disk is missing on
sysconfig -r
.
Pool0 spare disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
Spare disks for block checksum
spare 0a.10.3P2 0a 10 3 SA:B 0 FSAS 7200 55176/113000448 55184/113016832
spare 0a.10.4P2 0a 10 4 SA:B 0 FSAS 7200 55176/113000448 55184/113016832
spare 0a.10.6P2 0a 10 6 SA:B 0 FSAS 7200 55176/113000448 55184/113016832
spare 0a.10.7P2 0a 10 7 SA:B 0 FSAS 7200 55176/113000448 55184/113016832
spare 0a.10.8P2 0a 10 8 SA:B 0 FSAS 7200 55176/113000448 55184/113016832
spare 0a.10.10P2 0a 10 10 SA:B 0 FSAS 7200 55176/113000448 55184/113016832 (not zeroed)
spare 0a.10.11P2 0a 10 11 SA:B 0 FSAS 7200 55176/113000448 55184/113016832
spare 0b.00.0P2 0b 0 0 SA:A 0 FSAS 7200 55176/113000448 55184/113016832
spare 0b.00.5P2 0b 0 5 SA:A 0 FSAS 7200 55176/113000448 55184/113016832
spare 0b.00.11P2 0b 0 11 SA:A 0 FSAS 7200 55176/113000448 55184/113016832
spare 0a.10.5 0a 10 5 SA:B 0 FSAS 7200 9324290/19096145920 9342976/19134414848 (not zeroed)
- So the system start disk partition on another spare disk.
[?] Mon Apr 01 14:46:55 +0900 [node-01: config_thread: raid.partition.disk:notice]: Disk partition successful on Disk 0a.10.5 Shelf 10 Bay 5 [NETAPP X380_HLBRE10TA07 NA03] S/N [XXXXXXXX] UID [5000CCA2:5172ED2C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [1]=2372885888.
[?] Mon Apr 01 14:46:55 +0900 [node-01: disk_layout_admin: disk.layout.changed:debug]: Detected partition layout change on disk 0a.10.5.
[?] Mon Apr 01 14:46:55 +0900 [node-01: rastrace_dump: rastrace.dump.saved:debug]: A RAS trace dump for module RAID instance 0 was stored in /etc/log/rastrace/RAID_0_20240401_14:46:55:336307.dmp.
[?] Mon Apr 01 14:46:55 +0900 [node-01: dmgr_thread: raid.disk.inserted:info]: Disk 0a.10.5 Shelf 10 Bay 5 [NETAPP X380_HLBRE10TA07 NA03] S/N [XXXXXXXX] UID [5000CCA2:5172ED2C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] has been inserted into the system
- Trigger
SPARES_LOW
Autosupport message
[?] Mon Apr 01 14:46:56 +0900 [node-01: config_thread: raid.rg.spares.low:error]: /aggr01_02/plex0/rg0
[?] Mon Apr 01 14:46:56 +0900 [node-01: config_thread: callhome.spares.low:error]: Call home for SPARES_LOW: /aggr01_02/plex0/rg0
- The system picked up the first partitioned spare disk for reconstruction.
[?] Mon Apr 01 14:48:21 +0900 [node-01: config_thread: raid.rg.recons.info:notice]: Spare disk 0a.10.2P1 will be used to reconstruct one missing disk in RAID group /aggr01_02/plex0/rg0.
- unpartitioned the second spare disk.
[?] Mon Apr 01 14:51:13 +0900 [node-01: config_thread: raid.unpartition.disk:notice]: Disk unpartition successful on Disk 0a.10.5 Shelf 10 Bay 5 [NETAPP X380_HLBRE10TA07 NA03] S/N [XXXXXXXX] UID [5000CCA2:5172ED2C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000].
[?] Mon Apr 01 14:51:13 +0900 [node-01: disk_layout_admin: disk.layout.changed:debug]: Detected partition layout change on disk 0a.10.5.