High Worker Node CPU Usage Caused by Trident Orphaned Volumes Referencing Deleted Backend
Applies to
-
NetApp Astra Trident CSI Driver
-
Kubernetes / OpenShift environments
-
ONTAP SAN/NAS backends
Issue
Worker nodes running Trident CSI controller pods exhibit sustained high CPU utilization (up to 100%).
leading to:
- Cluster-wide performance degradation
- Application disruption across multiple pods
- Increased latency in provisioning and deletion workflows
Observed Log Patterns:
From Trident controller logs:
level=error msg="Invalid backend state." expectedState=online/deleting state=failed workflow="core=node_reconcile"msg="Unable to delete snapshot from backend."error="backend <backend-name> is not Online or Deleting"msg="Unable to delete volume from backend."error="backend <backend-name> is not Online or Deleting"- These errors repeat continuously (high frequency retries)
- Controller enters a persistent reconcile loop
