Replicated wafl.raid.incons.userdata messages after raid_multierr_bad_block_1 errors in SnapMirror or SnapVault clusters
Applies to
- ONTAP
- Production cluster with SapMirror and/or SnapVault destination clusters
- Replicated block with user data inconsistency
Issue
- WAFL user data inconsistency block messages in SnapMirror & SnapVault destination nodes with same file block number (fbn) and fileid. Example:
Fri Sep 17 06:23:59 CEST [SnapMirror_node1: wafl_exempt02: wafl.raid.incons.userdata:error]: WAFL inconsistent: inconsistent user data block at VBN 1792908851 (vvbn:10473883 fbn:22189 level:0) in public inode (fileid:12103 snapid:0 file_type:1 disk_flags:0x8002 error:120 raid_set:0) in volume mirror_volume@vserver:mmmmmmmm-mmmm-mmmm-mmmm-mmmmmmmmmmmm.
Fri Sep 17 08:22:04 CEST [SnapVault_node1: wafl_exempt00: wafl.raid.incons.userdata:error]: WAFL inconsistent: inconsistent user data block at VBN 3028359178 (vvbn:5407137 fbn:22189 level:0) in public inode (fileid:12103 snapid:0 file_type:1 disk_flags:0x8002 error:120 raid_set:0) in volume vault_volume@vserver:vvvvvvvv-vvvv-vvvv-vvvv-vvvvvvvvvvvv.
- The inconsistency is present in the source/production node for the same fbn and fileid:
Sat Sep 18 00:10:06 CEST [Production_node1: wafl_exempt01: wafl.raid.incons.userdata:error]: WAFL inconsistent: inconsistent user data block at VBN 13045486183 (vvbn:11602768 fbn:22189 level:0) in public inode (fileid:12103 snapid:0 file_type:1 disk_flags:0x2 error:120 raid_set:1) in volume production_volume@vserver:pppppppp-pppp-pppp-pppp-pppppppppppp.
- Previous bad RAID multierror block for that Virtual Block Number (VBN):
Fri Sep 17 05:00:20 CEST [Production-node1: raidio_thread: raid_multierr_bad_block_1:error]: params: {'disk_rpm': '10000', 'vendor': 'NETAPP ', 'firmware_revision': 'NA02', 'shelf': '10', 'disk_info': 'Disk /Production_node1_data_aggregate/plex0/rg3/0a.10.17 Shelf 10 Bay 17 [NETAPP X425_HCBEP1T2A10 NA02] S/N [12345678] UID [5000CCA0:1D9C8D88:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000]', 'volumeBno': '13045486183', 'site': 'Local', 'bay': '17', 'carrier': '', 'serialno': 'KZJT31RH', 'owner': '', 'model': 'X425_HCBEP1T2A10', 'disk_type': '4', 'blockNum': '178266599'}