C3 Tx discards and Frame timeout due to faulty SFP on end device

Applies to

• Brocade Fabric OS 
• SAN
• ONTAP 9 and above

Issue

• errdump -a logs  frame timeout, stating which port received a frame (rx) and where it cannot be transmitted on (tx) along with loss signal and oversubscribed alerts.

2022/02/09-22:31:04, [AN-1014], 2266, SLOT 2 | FID 128, INFO, switch, Frame timeout detected, tx port 9/7 rx port 9/27, sid c8f907, did 678740, timestamp 2022-02-09 22:31:04 .
2023/12/23-10:47:16 (IST), [MAPS-1003], 274485, SLOT 2 | FID 128 | PORT 11/8, WARNING, switch, IB_INFINI_3185_W_N3P6, F-Port 11/8, Condition=ALL_PORTS(PORT_BANDWIDTH/NONE==OVERSUBSCRIBED), Current Value:[PORT_BANDWIDTH, OVERSUBSCRIBED, (TXQL=914 us, TX=72.8%) ], RuleName=defALL_PORTS_OVERSUBSCRIBED, Dashboard Category=Fabric Performance Impact, Quiet Time=15 min.
2023/12/18-01:35:43 (IST), [MAPS-1003], 270764, SLOT 1 | FID 128 | PORT 10/21, WARNING, switch, U-Port 10/21, Condition=ALL_PORTS(LOSS_SIGNAL/min>5), Current Value:[LOSS_SIGNAL, 8 LOS], RuleName=defALL_PORTSLOSS_SIGNAL_5, Dashboard Category=Port Health, Quiet Time=None.

• The frame timeouts indicate that those Tx ports were unable to forward the frames across to the designated Rx ports and, hence got timed out.
• IO_FRAME_LOSS  events indicating frame delay and IO_PERF_IMPACT events logged in errdump on switch end -

2024/12/02-04:56:04 (IST), [MAPS-1001], 1664654, SLOT 2 | FID 128 | PORT 12/8, CRITICAL, switch, slot12 port8, F-Port 12/8, Condition=ALL_PORTS(DEV_LATENCY_IMPACT/NONE==IO_FRAME_LOSS), Current Value:[DEV_LATENCY_IMPACT, IO_FRAME_LOSS, (174 ms Frame Delay in VC: 2) ], RuleName=ALL_PORTS_IO_FRAME_LOSS_UNQUAR, Dashboard Category=Fabric Performance Impact, Quiet Time=1 day.

2024/12/02-04:56:04 (IST), [MAPS-1001], 1664654, SLOT 2 | FID 128 | PORT 12/8, CRITICAL, switch, slot12 port8, F-Port 12/8, Condition=ALL_PORTS(DEV_LATENCY_IMPACT/NONE==IO_FRAME_LOSS), Current Value:[DEV_LATENCY_IMPACT, IO_FRAME_LOSS, (174 ms Frame Delay in VC: 2) ], RuleName=ALL_PORTS_IO_FRAME_LOSS_UNQUAR, Dashboard Category=Fabric Performance Impact, Quiet Time=1 day.
2024/12/02-04:56:04 (IST), [MAPS-1003], 1664655, SLOT 2 | FID 128 | PORT 12/45, WARNING, switch, slot12 port45, F-Port 12/45, Condition=ALL_PORTS(DEV_LATENCY_IMPACT/NONE==IO_PERF_IMPACT), Current Value:[DEV_LATENCY_IMPACT, IO_PERF_IMPACT, (39.3% of 10 secs in VC: 3-7) ], RuleName=defALL_PORTS_IO_PERF_IMPACT_UNQUAR_1, Dashboard Category=Fabric Performance Impact, Quiet Time=1 day.

• Rx & Tx values are optimal on end device

Rx - 958.1 (uWatts)
Tx - 958.8 (uWatts)

• sfpshow reports both Tx and Rx power within the recommended range.

=============
Slot 12/Port 45:
=============
RX Power:    -2.7    dBm (532.0uW)
TX Power:    -1.5    dBm (711.2 uW)

• porterrshow shows link failures, link resets, c3 discards and Tx timeouts related error counter incrementing.

/fabos/cliexec/porterrshow:
          frames        enc     crc     crc     too     too     bad     enc    disc    link    loss    loss    frjt    fbsy     c3timeout     pcs      uncor
        tx       rx      in     err     g_eof   shrt    long    eof     out    c3      fail    sync    sig                      tx      rx      err     err
391:   45.3g    3.4g    0       0       0       0       0       0       0      43.1k    5       0       5       0       0      43.1k    0       0       2.5k

• c3-timeout tx:
  •  The number of transmit class 3 frames discarded at the transmission port due to timeout (platform- and port-specific).
  •  This indicates an issue with the device connected to the switch.
• sfpshow indicates low receive power

=============
Port  391:
=============
RX Power:    -8.1    dBm (155.8uW)
TX Power:    -3.0    dBm (500.6 uW)

• portshow indicates the remote device has reset the link more than the local port sent OFFLINE primitives out

portshow 391
[...]
Lr_in:             133     Ols_in:       5
Lr_out:            7       Ols_out:      6

• Multi RDY/Frame loss detected on the affected switch port:

2025/01/24-13:53:15, [C5-1040], 162569, SLOT 2 | CHASSIS, WARNING, Brocade_X7-8, Multi RDY/Frame Loss detected on Slot 12, Port 680(120) m_rdy(0x1)/m_frame(0x0). Link Reset done.
2025/01/24-13:53:16, [C5-1040], 162571, SLOT 2 | CHASSIS, WARNING, Brocade_X7-8, Multi RDY/Frame Loss detected on Slot 12, Port 680(120) m_rdy(0x1)/m_frame(0x0). Link Reset done.

• Below troubleshooting steps can be verified before proceeding with the solution.

  • Focus on analyzing the end device workload to perform hardware checks on the end device connected to the ports reporting timeouts and frame loss.

  • The following devices need to be verified and in case any issues observed on any of these, the same needs to be replaced on the end device:
    • SFP
    • Cable
    • HBA Card
    • Patch panel links and connectors defective.