Intermittent host reboot due to faulty server end sfp

Applies to

• Ontap 9
• RHEL
• FC
• Cisco

Issue

• RHEL host reboots intermittently with below events and errors:

Nov 17 15:41:39 host multipathd: asm!.asm_ctl_vmb: add path (uevent)
Nov 17 15:41:39 host multipathd: asm/.asm_ctl_vmb: failed to get path uid
Nov 17 15:41:39 host multipathd: uevent trigger error
Nov 17 15:41:39 host multipathd: asm!.asm_ctl_vbg5: add path (uevent)
Nov 17 15:41:39 host multipathd: asm/.asm_ctl_vbg5: failed to get path uid
Nov 17 15:41:39 host multipathd: uevent trigger error
Nov 17 15:10:01 host systemd: Removed slice User Slice of root.
Nov 17 15:10:37 host systemd-udevd: worker [113970] /devices/virtual/block/dm-8 is taking a long time
Nov 17 15:10:37 host systemd-udevd: worker [113971] /devices/virtual/block/dm-61 is taking a long time
Nov 17 15:10:37 host systemd-udevd: worker [113972] /devices/virtual/block/dm-6 is taking a long time

• Prior to host reboot, transport related errors are logged under var/log/messages:

Nov 17 15:09:31 host kernel: sd 1:0:4:48: [sdlu] tag#1 CDB: Test Unit Ready 00 00 00 00 00 00
Nov 17 15:09:31 host kernel: sd 1:0:4:49: [sdlx] tag#22 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=0s
Nov 17 15:09:31 host kernel: sd 1:0:4:49: [sdlx] tag#22 CDB: Test Unit Ready 00 00 00 00 00 00
Nov 17 15:09:36 host kernel: sd 1:0:2:7: rejecting I/O to offline device
Nov 17 15:09:36 host kernel: sd 1:0:2:7: [sdda] killing request
Nov 17 15:09:36 host kernel: sd 1:0:2:31: [sdeg] killing request
Nov 17 15:09:36 host kernel: sd 1:0:2:31: [sdeg] killing request
Nov 17 15:09:36 host kernel: sd 1:0:2:7: [sdda] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=5s
Nov 17 15:09:36 host kernel: sd 1:0:2:7: [sdda] CDB: Write(16) 8a 00 00 00 00 00 8d 0f e3 87 00 00 00 20 00 00
Nov 17 15:09:36 host kernel: blk_update_request: 5 callbacks suppressed
Nov 17 15:09:36 host kernel: blk_update_request: I/O error, dev sdda, sector 2366628743
Nov 17 15:09:36 host kernel: sd 1:0:2:31: [sdeg] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=5s
Nov 17 15:09:36 host kernel: sd 1:0:2:31: [sdeg] CDB: Write(16) 8a 00 00 00 00 00 01 f3 81 74 00 00 00 16 00 00

• These errors are impacted all of the available multipath paths leaving  "0 paths remaining" to the storage LUN's, which in turn is causing IO to completely fail to the target LUN's:

Nov 17 15:09:36 host multipathd: sdah: mark as failed
Nov 17 15:09:36 host multipathd: xxx: remaining active paths: 3
Nov 17 15:09:36 host multipathd: sdcj: mark as failed
Nov 17 15:09:36 host multipathd: xxx: remaining active paths: 2
Nov 17 15:09:36 host multipathd: sdot: mark as failed
Nov 17 15:09:36 host multipathd: xxx: remaining active paths: 1
Nov 17 15:09:36 host multipathd: sdux: mark as failed
Nov 17 15:09:36 host multipathd: xxx: remaining active paths: 0
Nov 17 15:09:36 host multipathd: sdn: mark as failed

• No performance issues seen on storage end.
• There are no such erroneous events logged during or before the issue time in storage EMS.
• On Cisco SAN switch,  the host connected interfaces experienced a loss of signal during the incident.
• Upon reviewing the onboarding logs,  incremental Rx counters were noticed on the host connected ports, which suggests that the end devices connected  need to be examined further.
• On switch end, the flogi database indicates that host connected interface does not seem to be connected to the switch, which indicates an issue at physical layer for that path from host to switch.
• Perform physical connectivity check including cable testing, patch panel testing and server end sfp.