Unable to access CVO via SSH and BlueXP showing "Failed" status
Applies to
- NetApp BlueXP
- Cloud Volume ONTAP
- Amazon web service (AWS)
Issue
- Unable to access CVO via SSH intermittently
- BlueXP showing "Failed" status of CVO
- Ping CVO cluster management IP has no problem
- Connector
server.logdetects following error frequently:
2025-06-09 09:50:08,032 UTC WARN [Update all CVOs statuses ] [xxxx] [ ] [System ] (oncloud-akka.actor.default-dispatcher-163) [AwsVsaWorkingEnvironmentStatusOperations:25] Cloud Manager cannot communicate with Cloud Volumes ONTAP because there is no connectivity or because the Cloud Volumes ONTAP system is not available.
com.netapp.oncloud.simplicator.client.common.SimplicatorBadRequestException: 500 Internal Server Error
- CVO shows the following error:
EMS.log
[?] Thu Jun 12 09:54:15 +0000 [xxxx-01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table rastrace_asup_dump"
sktrace.log
2025-06-12T09:52:38Z 72426422444479285 [4:0] STORAGE_SHELF_INFO: kern_shelf_dsa_populateNextCache: Table stack_db is empty - using DSA
2025-06-12T09:52:38Z 72426422444481389 [4:0] STORAGE_SHELF_ERR: kern_shelf_dsa_populateNextCache: DSA not ready
2025-06-12T09:52:38Z 72426422444492807 [4:0] KSMF_SMF_SVC_NORM: process_request: Processing for table kern_shelf took 10319 msec which is longer than the client's timeout of 5000
- BSD layer OPS histograms also showed the long latency records
OP/ms < 10ms < 20ms < 30ms < 50ms < 100ms < 200ms < 250ms < 300ms < 350ms < 400ms < 450ms < 500ms < 1s < 2s < 3s < 5s < 10s < 20s < 30s < 60s < 90s < 120s > 120s
- Access 36487211 136810 2322 4172 3917 2538 355 20 13 2 2 0 0 0 0 0 0 0 0 0 0 0 0
hi-pri 567516 2329 9 18 13 3 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
clusfs 12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
- Close 68046323 325220 10934 17139 17639 5149 320 120 71 28 21 13 20 2 0 0 0 0 0 0 0 0 0