Trident leaves POD in ContainerWaiting for provisioning ANF NFS PVs
Applies to
- Azure NetApp Files (ANF) with Standard networking feature
- Astra Trident for Openshift/Kubernetes from version 19.07 and higher
Issue
Using a newly setup ANF backend in Trident, the provisioning does not succeed and times out:
~$ oc get events
LAST SEEN TYPE REASON OBJECT MESSAGE
89s Warning FailedMount pod/nginx-pod-1 Unable to attach or mount volumes: unmounted volumes=[volume],
unattached volumes=[volume kube-api-access-7fthh]: timed out waiting for the condition
51s Warning FailedAttachVolume pod/nginx-pod-1 AttachVolume.Attach failed for volume "pvc-27468699-089f-4a47-8d9e-4a7c59e12e5c" : timed out waiting for external-attacher of csi.trident.netapp.io CSI driver to attach volume pvc-27468699-089f-4a47-8d9e-4a7c59e12e5c
When looking at a single claim UUID, the following can be seen in the events of the trident-controller-sidecar-csi-provisioner:
(a) the initial request took ten seconds for the response to arrive:I0410 18:30:23.854476 1 connection.go:184] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-1"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-2"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-3"}}],"requisite":[{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-1"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-2"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-3"}}]},"capacity_range":{"required_bytes":21474836480},"name":"pvc-23b8d581-52f3-467f-9801-febc99bfc13e","parameters":{"fsType":"nfs","selector":"performance=bronze"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"nfs"}},"access_mode":{"mode":5}}]}
I0410 18:30:33.965371 1 connection.go:187] GRPC error: rpc error: code = DeadlineExceeded desc = volume state is Creating, not Succeeded
W0410 18:30:33.965464 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 0
E0410 18:30:33.965490 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is Creating, not Succeeded
(b) the first retry comes 9 seconds later:
W0410 18:30:42.209074 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 1
E0410 18:30:42.209101 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
(c) The second retry after 16 more seconds:
W0410 18:30:58.463242 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 2
E0410 18:30:58.463271 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
(d) And the rest every 30 seconds:
W0410 18:31:28.776946 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 3
E0410 18:31:28.776974 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:31:59.046562 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 4
E0410 18:31:59.046597 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:32:29.272501 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 5
E0410 18:32:29.272528 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:32:59.492813 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 6
E0410 18:32:59.492842 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:33:29.753260 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 7
E0410 18:33:29.753286 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:34:00.004575 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 8
E0410 18:34:00.004604 1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:34:30.252470 1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 9