SAS adapter port PHYs disabled all except one after shelf added
Applies to
- Disk shelf DS2246 added to Metrocluster
- Disk model X356_TPM4V3T8AME
Issue
- Added disk shelf containing disk models X357_TPM4V3T8AME
- Multiple disks show as failed
62.0 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX)
62.1 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX) (Failed)
62.2 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX) (Failed)
62.3 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX) (Failed)
62.4 : NETAPP X356_TPM4V3T8AME NA03 0.0GB 0B/ (Failed-Unsupported)
62.5 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX) (Failed)
62.6 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX) (Failed)
62.7 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX)
62.8 : NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX) (Failed)
62.9 : NETAPP X356_TPM4V3T8AME NA03 0.0GB 0B/sect (Failed-Unsupported)
62.10: NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX)
62.11: NETAPP X356_TPM4V3T8AME NA03 3662.5GB 520B/sect (XXXXXXXXXXXX) (Failed)
- Disks were reseated but still appeared as failed
- All SAS ports on the adapter show as having disabled PHYs, including ports that aren't connected to the newly added shelf
Cluster::*> storage port show -node Node-01 *
Cable Length: 2m
Cable End Identifier: end_1
Cable Identifier: 500a09800856a299-500a09800e4055f4
Port Speed: 12 Gb/s
Port State: enabled
Port Status: online-degraded
Error Type: online-degraded
Error Severity: Error
Error Text: The port is in online degraded state.
Corrective Action: Check cable connections.
Phy State: [0] disabled, offline, 0 Gb/s
[1] disabled offline, 0 Gb/s
[2] disabled, offline, 0 Gb/s
[3] enabled, online, 12 Gb/s
- Cables, IOMs reseated but issue persists
- Takeover / Giveback performed, disks remain failed (Failed-Unsupported drives don't appear failed any longer)
- Failed drives were replaced however errors continue
[Node-01: disk_server_0: disk.ioMediumError:notice]: Medium error on disk 0b.62.3: op 0x28:00000010:0008 sector 16 SCSI:medium error - Unrecovered read error - If the disk is in a RAID group, the subsystem will attempt to reconstruct unreadable data (3 11 ff 0) (4492) Disk 0b.62.3 Shelf 62 Bay 3 [NETAPP X356_TPM4V3T8AME NA03] S/N [XXXXXXXXXXXX] UID [50000397:9C902830:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000]
[Node-01: pmcsas_timeout_0: sas.device.quiesce:debug]: Adapter 0a encountered a command timeout on disk device 0c.62.1. Quiescing the device.
[Node-01: pmcsas_timeout_0: sas.device.quiesce:debug]: Adapter 0a encountered a command timeout on disk device 0b.62.3. Quiescing the device.
[Node-01: pmcsas_timeout_0: sas.device.quiesce:debug]: Adapter 0a encountered a command timeout on disk device 0b.62.11. Quiescing the device.