CONTAP-46582: RPANIC during SnapMirror Replication

Issue

-Node panics with:
[RPANIC:page fault (supervisor read data, page not present) on VA 0x3e cs:rip 0x20:0xffffffff80abfc1a rflags 0x10206 in process NwkThd_01 on release 9.10.1P15 (C)]

• EMS logs we see Network Integrity check failure around the time of the panic:
  [Sun Feb 18 20:00:01 -0600 [node8: repl_Handle_reg: repl.engine.error:debug]: params: {'replStatus': '309', 'replFailureMsg': '5898803', 'replFailureMsgDetail': '0', 'functionName': 'void repl_spinnp::NetworkIntegrityChecker::validateIntegrity(repl_core::ReplOpData &)', 'lineNumber': '1190']
  

[Sun Feb 18 20:00:01 -0600 [node8: repl_Handle_reg: repl.engine.error:debug]: params: {'replStatus': '309', 'replFailureMsg': '5898803', 'replFailureMsgDetail': '0', 'functionName': 'void repl_spinnp::NetworkIntegrityChecker::validateIntegrity(repl_core::ReplOpData &)', 'lineNumber': '1246'}
Sun Feb 18 20:00:05 -0600 [node8: rastrace_dump: rastrace.dump.saved:debug]: A RAS trace dump for module REPL instance 0 was stored in /etc/log/rastrace/REPL_0_20240218_20:00:05:649324.dmp.
Sun Feb 18 20:00:06 -0600 [node8: repl_Handle_reg: repl.engine.error:debug]: params: {'replStatus': '42', 'replFailureMsg': '5898536', 'replFailureMsgDetail': '4194309', 'functionName': 'spinnp_repl_result_t repl_core::ReplOpData::unmarshal()', 'lineNumber': '864'}
Sun Feb 18 20:01:02 -0600 [4294967295@node8: NwkThd_01: ha.panicInfoNotSent:notice]: Local node cannot send a panic information message to the HA partner over the HA interconnect due to error: Not HA, or not ESX/MVRD layer 3. Partner name: node7. Partner system ID: 537410192.
Sun Feb 18 20:01:02 -0600 [4294967295@node8: NwkThd_01: sk.panic:alert]: Panic String: page fault (supervisor read data, page not present) on VA 0x3e cs:rip 0x20:0xffffffff80abfc1a rflags 0x10206 in process NwkThd_01 on release 9.10.1P15 ]

• In SnapMirror Audit logs we see the integrity check fail due to network issues:
  [Sun Feb 18 20:00:01 CST 2024 ScheduledUpdate[Feb 18 20:00:00]:5e691939-252a-11ec-b835-00a098a77bf5 Operation-Uuid=97a5c86f-cc08-4dce-b611-8c373b9606a5 Group=none Operation-Cookie=0 action=Defer source=src_svm:vol1 destination=dst_svm:vol1_mirror status=Failure message=Transfer failed.(Replication operation integrity check failed due to transmission glitch or network issue.)]