Phantom hard drive found on SolidFire storage node

Applies to

H300S

Issue

A phantom failed drive is discovered on the storage node. When logs were collected and uploaded to NetApp Support for review, the following results were found:

The kern.log reports:

2025-04-28T07:03:27.913395Z stor-a rsyslogd: -- MARK --
2025-04-28T07:23:27.989393Z stor-a rsyslogd: -- MARK --
2025-04-28T07:32:36.001387Z stor-a kernel: [383399.103129] sd 0:0:6:0: attempting task abort! scmd(ffff893131146d00)
2025-04-28T07:32:36.001398Z stor-a kernel: [383399.103134] sd 0:0:6:0: [sdc] tag#0 CDB: ATA command pass through(16) 85 08 0e 00 d0 00 01 00 00 00 4f 00 c2 40 b0 00
2025-04-28T07:32:36.001400Z stor-a kernel: [383399.103136] scsi target0:0:6: handle(0x000b), sas_address(0x4433221102000000), phy(2)
2025-04-28T07:32:36.001404Z stor-a kernel: [383399.103138] scsi target0:0:6: enclosure_logical_id(0x5003048020897911), slot(2)
2025-04-28T07:32:36.001411Z stor-a kernel: [383399.103139] scsi target0:0:6: enclosure level(0x0000),connector name(    )
2025-04-28T07:32:39.913394Z stor-a kernel: [383403.016661] sd 0:0:6:0: task abort: SUCCESS scmd(ffff893131146d00)
2025-04-28T07:33:13.161088Z stor-a kernel: [383436.267461] mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
2025-04-28T07:33:16.411245Z stor-a kernel: [383439.517677] sd 0:0:6:0: device_block, handle(0x000b)
2025-04-28T07:33:18.661391Z stor-a kernel: [383441.767241] sd 0:0:6:0: device_unblock and setting to running, handle(0x000b)
2025-04-28T07:33:18.665383Z stor-a kernel: [383441.768518] sd 0:0:6:0: [sdc] Synchronizing SCSI cache
2025-04-28T07:33:18.665391Z stor-a kernel: [383441.768539] sd 0:0:6:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
2025-04-28T07:33:18.765386Z stor-a kernel: [383441.868155] mpt3sas_cm0: removing handle(0x000b), sas_addr(0x4433221102000000)
2025-04-28T07:33:18.765400Z stor-a kernel: [383441.868156] mpt3sas_cm0: removing : enclosure logical id(0x5003048020897911), slot(2)
2025-04-28T07:33:18.765402Z stor-a kernel: [383441.868157] mpt3sas_cm0: removing enclosure level(0x0000), connector name(     )
2025-04-28T07:35:25.913394Z stor-a kernel: [383569.020774] scsi 0:0:7:0: Direct-Access     ATA      SAMSUNG MZ7LM480 404Q PQ: 0 ANSI: 6
2025-04-28T07:35:25.913409Z stor-a kernel: [383569.020784] scsi 0:0:7:0: SATA: handle(0x000b), sas_addr(0x4433221102000000), phy(2), device_name(0x0000000000000000)
2025-04-28T07:35:25.913413Z stor-a kernel: [383569.020787] scsi 0:0:7:0: SATA: enclosure_logical_id(0x5003048020897911), slot(2)
2025-04-28T07:35:25.913417Z stor-a kernel: [383569.020789] scsi 0:0:7:0: SATA: enclosure level(0x0000), connector name(     )
2025-04-28T07:35:25.913420Z stor-a kernel: [383569.020838] scsi 0:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
2025-04-28T07:35:25.953381Z stor-a kernel: [383569.058435] sd 0:0:7:0: Attached scsi generic sg2 type 0
2025-04-28T07:35:25.953389Z stor-a kernel: [383569.058735] sd 0:0:7:0: [sdc] 937703088 512-byte logical blocks: (480 GB/447 GiB)
2025-04-28T07:35:57.473390Z stor-a kernel: [383600.578712] sd 0:0:7:0: attempting task abort! scmd(ffff893136437000)
2025-04-28T07:35:57.473406Z stor-a kernel: [383600.578718] sd 0:0:7:0: tag#0 CDB: Mode Sense(6) 1a 00 3f 00 04 00
2025-04-28T07:35:57.473408Z stor-a kernel: [383600.578721] scsi target0:0:7: handle(0x000b), sas_address(0x4433221102000000), phy(2)
2025-04-28T07:35:57.473409Z stor-a kernel: [383600.578722] scsi target0:0:7: enclosure_logical_id(0x5003048020897911), slot(2)
2025-04-28T07:35:57.473412Z stor-a kernel: [383600.578723] scsi target0:0:7: enclosure level(0x0000),connector name(    )
2025-04-28T07:36:01.413393Z stor-a kernel: [383604.521702] sd 0:0:7:0: task abort: SUCCESS scmd(ffff893136437000)
2025-04-28T07:36:01.413404Z stor-a kernel: [383604.521717] sd 0:0:7:0: [sdc] Write Protect is off
2025-04-28T07:36:01.413409Z stor-a kernel: [383604.521720] sd 0:0:7:0: [sdc] Mode Sense: 00 00 00 00
2025-04-28T07:36:01.665381Z stor-a kernel: [383604.771326] sd 0:0:7:0: [sdc] Asking for cache data failed
2025-04-28T07:36:01.665390Z stor-a kernel: [383604.771328] sd 0:0:7:0: [sdc] Assuming drive cache: write through
2025-04-28T07:36:03.165390Z stor-a kernel: [383606.271356] sd 0:0:7:0: [sdc] Spinning up disk...
2025-04-28T07:36:34.337392Z stor-a kernel: [383637.443371] sd 0:0:7:0: attempting task abort! scmd(ffff893136437000)
2025-04-28T07:36:34.337409Z stor-a kernel: [383637.443377] sd 0:0:7:0: tag#0 CDB: Start/Stop Unit 1b 01 00 00 01 00
2025-04-28T07:36:34.337411Z stor-a kernel: [383637.443381] scsi target0:0:7: handle(0x000b), sas_address(0x4433221102000000), phy(2)
2025-04-28T07:36:34.337416Z stor-a kernel: [383637.443383] scsi target0:0:7: enclosure_logical_id(0x5003048020897911), slot(2)
2025-04-28T07:36:34.337419Z stor-a kernel: [383637.443385] scsi target0:0:7: enclosure level(0x0000),connector name(    )
2025-04-28T07:36:37.912233Z stor-a kernel: [383641.022259] sd 0:0:7:0: device_block, handle(0x000b)
2025-04-28T07:36:38.165384Z stor-a kernel: [383641.272687] sd 0:0:7:0: task abort: SUCCESS scmd(ffff893136437000)
2025-04-28T07:36:39.665387Z stor-a kernel: [383642.295455] .
2025-04-28T07:36:39.665396Z stor-a kernel: [383642.772286] sd 0:0:7:0: device_unblock and setting to running, handle(0x000b)
2025-04-28T07:36:39.665398Z stor-a kernel: [383642.772308] ready
2025-04-28T07:36:39.665400Z stor-a kernel: [383642.772372] sd 0:0:7:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
2025-04-28T07:36:39.665417Z stor-a kernel: [383642.772375] sd 0:0:7:0: [sdc] Sense not available.
2025-04-28T07:36:39.665419Z stor-a kernel: [383642.772433] sd 0:0:7:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
2025-04-28T07:36:39.665420Z stor-a kernel: [383642.772435] sd 0:0:7:0: [sdc] Sense not available.
2025-04-28T07:36:39.665421Z stor-a kernel: [383642.772562] sdc: detected capacity change from 480103981056 to 0
2025-04-29T22:30:22.023879Z stor-a kernel: imklog 5.8.6, log source = /proc/kmsg started.

The sf-master.error log reports

2025-04-28T07:33:22.474625Z stor-a master-1[18570]: [Drive] 18956 GlobalPool-0 drive/DriveDevice.cpp:1139:TryReadLink|stat failed on result=/dev/slot2
2025-04-28T07:33:22.474742Z stor-a master-1[18570]: [Drive] 18956 GlobalPool-0 drive/Fstab.cpp:486:GetConnectedDevices|Missing dev.driveType=Block ent->path=/dev/slot2 dev={"canonicalName":"","connected":false,"dev":0,"devPath":"/dev/slot2","driveType":"Block","name":"slot2","path":"/dev/slot2","pathLink":"/dev/slot2","product":"Unknown","scsiCompatId":"","scsiState":"Unknown","securityAtMaximum":false,"securityEnabled":false,"securityFrozen":false,"securityLocked":false,"securitySupported":false,"serial":"","size":0,"slot":2,"uuid":"d41d8cd9-8f00-b204-e980-0998ecf8427e","vendor":"Unknown","version":"Unknown"}
2025-04-28T07:33:24.383066Z stor-a master-1[18570]: [Drive] 18956 GlobalPool-0 drive/DriveDevice.cpp:1139:TryReadLink|stat failed on result=/dev/slot2
2025-04-28T07:33:24.383178Z stor-a master-1[18570]: [Drive] 18956 GlobalPool-0 drive/Fstab.cpp:486:GetConnectedDevices|Missing dev.driveType=Block ent->path=/dev/slot2 dev={"canonicalName":"","connected":false,"dev":0,"devPath":"/dev/slot2","driveType":"Block","name":"slot2","path":"/dev/slot2","pathLink":"/dev/slot2","product":"Unknown","scsiCompatId":"","scsiState":"Unknown","securityAtMaximum":false,"securityEnabled":false,"securityFrozen":false,"securityLocked":false,"securitySupported":false,"serial":"","size":0,"slot":2,"uuid":"d41d8cd9-8f00-b204-e980-0998ecf8427e","vendor":"Unknown","version":"Unknown"}
2025-04-28T07:33:26.294784Z stor-a master-1[18570]: [Drive] 18956 GlobalPool-0 drive/DriveDevice.cpp:1139:TryReadLink|stat failed on result=/dev/slot2
2025-04-28T07:33:26.294899Z stor-a master-1[18570]: [Drive] 18956 GlobalPool-0 drive/Fstab.cpp:486:GetConnectedDevices|Missing dev.driveType=Block ent->path=/dev/slot2 dev={"canonicalName":"","connected":false,"dev":0,"devPath":"/dev/slot2","driveType":"Block","name":"slot2","path":"/dev/slot2","pathLink":"/dev/slot2","product":"Unknown","scsiCompatId":"","scsiState":"Unknown","securityAtMaximum":false,"securityEnabled":false,"securityFrozen":false,"securityLocked":false,"securitySupported":false,"serial":"","size":0,"slot":2,"uuid":"d41d8cd9-8f00-b204-e980-0998ecf8427e","vendor":"Unknown","version":"Unknown"}