Skip to main content
NetApp Knowledge Base

Trident leaves POD in ContainerWaiting for provisioning ANF NFS PVs

Views:
195
Visibility:
Public
Votes:
0
Category:
trident-kubernetes
Specialty:
snapx
Last Updated:

Applies to

  • Azure NetApp Files (ANF) with Standard networking feature
  • Astra Trident for Openshift/Kubernetes from version 19.07 and higher

Issue

Using a newly setup ANF backend in Trident, the provisioning does not succeed and times out:

~$ oc get events
LAST SEEN   TYPE      REASON               OBJECT            MESSAGE
89s         Warning   FailedMount          pod/nginx-pod-1   Unable to attach or mount volumes: unmounted volumes=[volume],
unattached volumes=[volume kube-api-access-7fthh]: timed out waiting for the condition

51s         Warning   FailedAttachVolume   pod/nginx-pod-1   AttachVolume.Attach failed for volume "pvc-27468699-089f-4a47-8d9e-4a7c59e12e5c" : timed out waiting for external-attacher of csi.trident.netapp.io CSI driver to attach volume pvc-27468699-089f-4a47-8d9e-4a7c59e12e5c


When looking at a single claim UUID, the following can be seen in the events of the trident-controller-sidecar-csi-provisioner:

(a) the initial request took ten seconds for the response to arrive:I0410 18:30:23.854476       1 connection.go:184] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-1"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-2"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-3"}}],"requisite":[{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-1"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-2"}},{"segments":{"topology.kubernetes.io/region":"westus3","topology.kubernetes.io/zone":"westus3-3"}}]},"capacity_range":{"required_bytes":21474836480},"name":"pvc-23b8d581-52f3-467f-9801-febc99bfc13e","parameters":{"fsType":"nfs","selector":"performance=bronze"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"nfs"}},"access_mode":{"mode":5}}]}

I0410 18:30:33.965371       1 connection.go:187] GRPC error: rpc error: code = DeadlineExceeded desc = volume state is Creating, not Succeeded

W0410 18:30:33.965464       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 0
E0410 18:30:33.965490       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is Creating, not Succeeded

(b) the first retry comes 9 seconds later:

W0410 18:30:42.209074       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 1
E0410 18:30:42.209101       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded

 

(c) The second retry after 16 more seconds:

W0410 18:30:58.463242       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 2
E0410 18:30:58.463271       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded

(d) And the rest every 30 seconds:
W0410 18:31:28.776946       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 3
E0410 18:31:28.776974       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:31:59.046562       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 4
E0410 18:31:59.046597       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:32:29.272501       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 5
E0410 18:32:29.272528       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:32:59.492813       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 6
E0410 18:32:59.492842       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:33:29.753260       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 7
E0410 18:33:29.753286       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:34:00.004575       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 8
E0410 18:34:00.004604       1 controller.go:957] error syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e": failed to provision volume with StorageClass "azure-netapp-files-bronze": rpc error: code = DeadlineExceeded desc = volume state is still Creating, not Succeeded
W0410 18:34:30.252470       1 controller.go:934] Retrying syncing claim "23b8d581-52f3-467f-9801-febc99bfc13e", failure 9

No more requests come after the last failure, 4 minutes and 7 seconds after the initial creation request.

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.