NVMe disk missing on one controller during cluster setup
Applies to
- AFF A20
- AFF A90
- ONTAP 9
Issue
- Disk on slot 22 (or any other slot) missing during or after cluster setup as seen on maintenance mode
*> disk show -pLocal System ID: 12345678
DISK OWNER POOL SERIAL NUMBER ------------ ------------ ---- ------------- 0n.21 (00000000) Pool0 S12345677891011 0n.20 (00000000) Pool0 S71232435456671 0n.2 (00000000) Pool0 S71123254315213 0n.0 (00000000) Pool0 S71032483294231 0n.3 (00000000) Pool0 S74920384923921 0n.23 (00000000) Pool0 S71394032149032 0n.1 (00000000) Pool0 S73489392483291
- Same disk visible on other slots
*> disk show -pLocal System ID: 12345678
DISK OWNER POOL SERIAL NUMBER ------------ ------------ ---- ------------- 0n.21 (00000000) Pool0 S12345677891011 0n.20 (00000000) Pool0 S71232435456671 0n.2 (00000000) Pool0 S71123254315213 0n.0 (00000000) Pool0 S71032483294231 0n.3 (00000000) Pool0 S74920384923921 0n.23 (00000000) Pool0 S71394032149032 0n.1 (00000000) Pool0 S73489392483291 0n.19 (00000000) Pool0 S71312312312312
- Events reported on console during boot
Jun 01 14:08:41 [localhost:diskown.errorDuringIO:error]: error 3 (disk failed) on disk 374E3630:59A00526:00253854:00000002:00000000:00000000:00000000:00000000:00000000:00000000 (S/N N/A) while opening diskJun 01 14:08:41 [localhost:diskown.errorDuringIO:error]: error 3 (disk failed) on disk 374E3630:59A00473:00253854:00000002:00000000:00000000:00000000:00000000:00000000:00000000 (S/N N/A) while opening disk
Jun 02 15:31:34 [localhost:nvme.ssd.attach.error:error]: NVMe SSD encountered error: "ctrlr construct failed" on slot 22. Could not attach drive.Jun 02 15:31:39 [localhost:nvme.ssd.attach.error:error]: NVMe SSD encountered error: "ctrlr construct failed" on slot 22. Could not attach drive.
- If the node is up in ONTAP, the following may be seen in the nodeshell command output s
torage show shelf
Working node:
Device Port Negotiated Medium Device Port Negotiated ReceivedDisk Port Max Link Max Link Link Max Link Max Lane Max Lane Lane Error Bad TLP Bad DLLP Uncorrectable CorrectableId State Speed Speed Speed Speed Width width width Count Count Count Errors Errors (GT/s) (GT/s) (GT/s) (GT/s)------------------------------------------------------------------------------------------------------------------------------------------------------------------------------[ 0 ] OK 16.0 16.0 16.0 8.0 2 2 2 0 0 0 0 0[ 1 ] OK 16.0 16.0 16.0 8.0 2 2 2 0 0 0 0 0[ 2 ] OK 16.0 16.0 16.0 8.0 2 2 2 0 0 0 0 0[ 3 ] OK 16.0 16.0 16.0 8.0 2 2 2 0 0 0 0 0
Affected node:
Device Port Negotiated Medium Device Port Negotiated ReceivedDisk Port Max Link Max Link Link Max Link Max Lane Max Lane Lane Error Bad TLP Bad DLLP Uncorrectable CorrectableId State Speed Speed Speed Speed Width width width Count Count Count Errors Errors (GT/s) (GT/s) (GT/s) (GT/s)------------------------------------------------------------------------------------------------------------------------------------------------------------------------------[ 0 ] OK 16.0 16.0 16.0 8.0 2 2 2 0 0 0 0 0[ 1 ] OK 16.0 16.0 16.0 8.0 2 2 2 0 0 0 0 0[ 2 ] OK 16.0 16.0 16.0 8.0 2 2 2 0 0 0 0 0[ 3 ] DIS/DLL NA 16.0 2.5 8.0 0 2 0 0 0 0 0 0
- If seen during setup, swapping the nodes between slots of the chassis results in the issue follows the node
