Trident failed to create backend on OCP 4.14 Error: error communicating with Trident REST API
Applies to
- Astra Trident
- OpenShift Container Platform(OCP) 4.14 and above
Issue
Creating a backend fails with below error:
./tridentctl create backend -f trident-backend-tridentctl.yaml -n trident -d
DEBU[2024-08-12T13:25:47+12:00] >>>> command.ExecuteWithoutLog args="[get pod -n trident -l app=controller.csi.trident.netapp.io -o=json --field-selector=status.phase=Running]" command=oc
DEBU[2024-08-12T13:25:47+12:00] <<<< command.ExecuteWithoutLog command=oc error="<nil>"
Operating mode = tunnel, Trident pod = trident-controller-7cbf776784-gq7kj, Namespace = trident, CLI = oc
Invoking tunneled command: oc exec trident-controller-7cbf776784-gq7kj -n trident -c trident-main -- tridentctl --debug create backend --base64 eyJiYWNrZW5kTmFtZSI6InN2bS1vY3AtbWRyNWdjdGwwMSIsImRlYnVnVHJhY2VGbGFncyI6eyJhcGkiOnRydWUsIm1ldGhvZCI6dHJ1ZX0sIm1hbmFnZW1lbnRMSUYiOiIxMC4xNC4xMzQuMTEiLCJwYXNzd29yZCI6Ik9wMzRTaDE0NyIsInNhblR5cGUiOiJpc2NzaSIsInN0b3JhZ2VEcml2ZXJOYW1lIjoib250YXAtc2FuIiwic3ZtIjoic3ZtLW9jcC1tZHI1Z2N0bDAxIiwidXNlcm5hbWUiOiJ2c2FkbWluIiwidmVyc2lvbiI6MX0=
DEBU[2024-08-12T13:25:47+12:00] >>>> command.ExecuteWithoutLog args="[exec trident-controller-7cbf776784-gq7kj -n trident -c trident-main -- tridentctl --debug create backend --base64 eyJiYWNrZW5kTmFtZSI6InN2bS1vY3AtbWRyNWdjdGwwMSIsImRlYnVnVHJhY2VGbGFncyI6eyJhcGkiOnRydWUsIm1ldGhvZCI6dHJ1ZX0sIm1hbmFnZW1lbnRMSUYiOiIxMC4xNC4xMzQuMTEiLCJwYXNzd29yZCI6Ik9wMzRTaDE0NyIsInNhblR5cGUiOiJpc2NzaSIsInN0b3JhZ2VEcml2ZXJOYW1lIjoib250YXAtc2FuIiwic3ZtIjoic3ZtLW9jcC1tZHI1Z2N0bDAxIiwidXNlcm5hbWUiOiJ2c2FkbWluIiwidmVyc2lvbiI6MX0=]" command=oc
DEBU[2024-08-12T13:27:17+12:00] <<<< command.ExecuteWithoutLog command=oc error="exit status 1"
Operating mode = direct, Server = 127.0.0.1:8000, Autosuport server = 127.0.0.1:8003
Trident URL: http://127.0.0.1:8000/trident/v1
time="2024-08-12T01:25:47Z" level=debug msg="--------------------------------------------------------------------------------\n"
time="2024-08-12T01:25:47Z" level=debug msg="Request Method: POST\n"
time="2024-08-12T01:25:47Z" level=debug msg="Request URL: http://127.0.0.1:8000/trident/v1/backend\n"
time="2024-08-12T01:25:47Z" level=debug msg="Request headers: map[Content-Type:[application/json]]\n"
time="2024-08-12T01:25:47Z" level=debug msg="Request body: {\"backendName\":\"svm-ocp-mdr5gctl01\",\"debugTraceFlags\":{\"api\":true,\"method\":true},\"managementLIF\":\"10.14.134.11\",\"password\":<REDACTED>,\"sanType\":\"iscsi\",\"storageDriverName\":\"ontap-san\",\"svm\":\"svm-ocp-mdr5gctl01\",\"username\":<REDACTED>,\"version\":1}\n"
time="2024-08-12T01:25:47Z" level=debug msg="................................................................................\n"
Error: error communicating with Trident REST API; Post http://127.0.0.1:8000/trident/v1/backend: EOF
command terminated with exit code 1
Error: exit status 1
Trident Controller log:
level=error msg="Error creating ONTAP REST API client for initial call. Falling back to ZAPI." crdControllerEvent=update error="unable to get details for SVM <SVM name>; Get \https://<mgmt ip>/api/svm/svms?fields=&name=<SVM name>\: context deadline exceeded" logLayer=core requestID=75e0863b-d633-4cc0-adbb-e701604129b5 requestSource=CRD workflow="cr=reconcile"
level=error msg="API invocation failed. Post \https://<mgmt ip>/servlets/netapp.servlets.admin.XMLrequest_filer\: context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
level=error msg="Could not initialize storage driver." crdControllerEvent=update error="error initializing ontap-nas-economy driver: could not create Data ONTAP API client: error creating ONTAP API client: error reading SVM details: Post \https://<mgmt ip>/servlets/netapp.servlets.admin.XMLrequest_filer\: context deadline exceeded (Client.Timeout exceeded while awaiting headers)" logLayer=core requestID=75e0863b-d633-4cc0-adbb-e701604129b5 requestSource=CRD workflow="cr=reconcile"