CRC errors received on a single NIC port due to faulty switch end sfp
Applies to
- ONTAP 9
- FAS / AFF Systems
- CRC errors reporting on a single port
Issue
Event
log report hardware errors on a physical and/or logical port.
[node-01: vifmgr: vifmgr.cluscheck.crcerrors]: Port a0b on node node-01 is reporting a high number of observed hardware errors, possibly CRC errors
[node-02: vifmgr: vifmgr.cluscheck.crcerrors]: Port e0d on node node-02 is reporting a high number of observed hardware errors, possibly CRC errors
[node-02: vifmgr: vifmgr.cluscheck.hwerrors:alert]: Port e0d on node node-02 is reporting a high number (at least 1 per 1000 packets) of observed hardware errors (CRC, length, alignment, dropped)
[node-02: vifmgr: callhome.clus.net.degraded:alert]: Call home for CLUSTER NETWORK DEGRADED: CRC Errors Detected - High CRC errors detected on port e0d node node-02
ifstat
output showCRC
errors if ONTAP is receiving the errors.
RECEIVE
Total frames: 36418m | Frames/second: 23646 | Total bytes: 179t
Bytes/second: 116m | Total errors: 170k | Errors/minute: 7
Total discards: 0 | Discards/minute: 0 | Multi/broadcast: 1686k
Non-primary u/c: 0 | CRC errors: 159k | Long frames: 0
- CRC errors may be observed on a switch port or client side and latency may be seen due to packet loss
2022-03-20T17:39:36.443Z cpu36:2098075)WARNING: ScsiDeviceIO: 1498: Device naa.600a09803830574c4d5d53ddf26c4543 performance has deteriorated. I/O latency increased from average value of 18171 microseconds to 1816780 microseconds.
- Run
ifstat -z
on the NetApp end to clear the stats on the affected node and check for incrementing errors.
- Below steps can be followed to isolate the issue further-
- CRC errors indicate an issue with the media (such as a badly seated, broken cable and SFP )
- Perform a port isolation test to ensure port is in good health.
- Verify the health of the connection between the port reporting error and the next device in the link.
- Try reseating both ends of the cable.
- Toggle the port to see determine if issue is resolved
- Reseat the SFP of both ends (one at a time).
- Replace the connecting cable.
- If issue is not fixed after SFP or cable replacement, proceed with solution.