Aggregate mirroring degraded and plex offline
Applies to
- Metrocluster IP
- NetApp ONTAP 9.16.1P12
- NetApp AFF A400
Issue
- Aggregate is in a “mirror degraded” state with remote plexes showing as offline/failed/inactive
- EMS log reports
raid.vol.unprotected.remotesyncmirror: Aggregate aggr1 is mirrored and one plex is not online. The volume will not be available if a takeover occurs and the online plex is not accessible to the partner node.- Access to data is still available from the local plex
- Metrocluster interface e1a is down - IFSTAT counters showed CRC errors and link flapping
-- interface e1a (83 days, 3 hours, 51 minutes, 42 seconds) --RECEIVE Total frames: 102g | Frames/second: 0 | Total bytes: 284t Bytes/second: 0 | Total errors: 287 | Errors/minute: 0 Total discards: 1075 | Discards/minute: 0 | Multi/broadcast: 247k Non-primary u/c: 0 | CRC errors: 201 | Long frames: 0 Jabber: 86 | Length errors: 0 | No buffer: 1075 Pause: 248k | Jumbo: 34825m | Error symbol: 0 Bus overruns: 0 | LRO segments: 10001m | LRO bytes: 199t LRO6 segments: 0 | LRO6 bytes: 0 | Bad UDP cksum: 0 Bad UDP6 cksum: 0 | Bad TCP cksum: 0 | Bad TCP6 cksum: 0 Mcast v6 solicit: 0 | Lagg errors: 0 | Lacp errors: 0 Lacp PDU errors: 0TRANSMIT Total frames: 126g | Frames/second: 0 | Total bytes: 481t Bytes/second: 0 | Total errors: 0 | Errors/minute: 0 Total discards: 0 | Queue overflow: 0 | Multi/broadcast: 236k Collisions: 0 | Pause: 34051k | Jumbo: 58325m Cfg Up to Downs: 2 | TSO segments: 4194m | TSO bytes: 208t TSO6 segments: 0 | TSO6 bytes: 0 | HW UDP cksums: 20090k HW UDP6 cksums: 0 | HW TCP cksums: 14029m | HW TCP6 cksums: 0 Mcast v6 solicit: 5 | Lagg drops: 0 | Lagg no buffer: 0 Lagg no entries: 0DEVICE Mcast addresses: 6 | Rx MBuf Sz: 4096LINK INFO Speed: 0 | Duplex: full | Flowcontrol: full Media state: no carrier | Up to downs: 4626 | HW assist: 514k- IFCONFIG -vvv shows the link as administratively up, but we see there is 'no carrier' reported and the RX power is 0
e1a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=66ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP,VXLAN_HWCSUM,VXLAN_HWTSO> capabilities=66ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP,VXLAN_HWCSUM,VXLAN_HWTSO> ether 00:xx:xx:xx:xx:xx media: Ethernet autoselect (autoselect <full-duplex,rxpause,txpause>) status: no carrier supported media: media autoselect plugged: QSFP28 Unknown (MPO 1x12 Parallel Optic) vendor: FINISAR CORP PN: 332-00440 SN: XXXXXXXXXXX DATE: 2022-04-06 compliance level: SFF-8636 rev >=2.8 nominal bitrate: 25750 Mbps module temperature: 29.80 C voltage: 3.26 Volts lane 1: RX power: 0.00 mW (-40.00 dBm) TX bias: 8.41 mA lane 2: RX power: 0.00 mW (-40.00 dBm) TX bias: 8.41 mA lane 3: RX power: 0.00 mW (-40.00 dBm) TX bias: 8.41 mA lane 4: RX power: 0.00 mW (-40.00 dBm) TX bias: 8.29 mA