CONTAP-176582: Delay in disk failure undergoing reconstruction causes huge latency
Issue
When the disk undergoing raid reconstruction receives excessive media errors, ONTAP will take longer to fail the disk, compared to failing a disk that is not going through raid reconstruction. This is so far the designed behavior since ONTAP is trying to avoid any potential multiple disk failure scenario.
This delay in the disk layer will cause high Read/Write latency and long CPs.
This usually comes with the following errors in EMS:
1. [<node_name>: disk_server_0: shm.threshold.mediumErrors:error]: shm: Disk 9a.11.10 has crossed the medium error threshold in a 10 minute window.
2. [<node_name>: disk_server_0: scsi.debug:debug]: shm_setup_for_failure disk 9a.11.10 (S/N WBN6BQ3N) error 2h
3. [<node_name>: disk_server_0: scsi.debug:debug]: shm_setup_for_failure disk 9a.11.10 (S/N WBN6BQ3N) error 20h
4. [<node_name>: disk_latency_monitor: shm.threshold.highIOLatency:error]: Disk 9a.11.10 exceeds the average IO latency threshold and will be recommended for failure.
5. [<node_name>: disk_latency_monitor: scsi.debug:debug]: shm_setup_for_failure disk 9a.11.10 (S/N WBN6BQ3N) error 200000h
6. [<node_name>: wafl_exempt00: wafl.cp.toolong:error]: Aggregate <aggr_name> experienced a long CP.
7. [<node_name>: disk_server_0: scsi.debug:debug]: shm_setup_for_failure disk 9a.11.10 (S/N WBN6BQ3N) error 40000000h
Note: The following error codes are non-urgent error codes, ONTAP would not fail the disk immediately.
- 0x00000002
- 0x00000020
- 0x00200000