Skip to main content
NetApp Knowledge Base

CVO HA custer degraded because AWS mediator fails to update route tables

  1. Last updated
  2. Save as PDF
Views:
15
Visibility:
Public
Votes:
0
Category:
cloud-manager
Specialty:
cloud
Last Updated:

Applies to

  • NetApp Console
  • Cloud Volumes ONTAP (CVO)
  • Amazon Web Services (AWS)

Issue

During a DR exercise, the  AWS HA CVO enters a degraded state because the mediator is not able to update the AWS route tables for the floating IP addresses.
The NetApp Console UI reports:
“HA cluster is not highly available. Failed to establish HTTP connection to the mediator.”
Agent server logs shows:
2026-03-19 16:05:56,423 UTC ERROR [Working Environment           ] [85511a3f-62ab-4aeb-80d0-645648c21383] [H9PKZeTAlX   ] [User Name] (simplicator-akka.actor.default-dispatcher-328808) [SimplicatorServerActor:291] Simplicator request to http://localhost:8080/AwsProtocol/de...gion=us-west-2 timed out
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to ec2.us-west-2.amazonaws.com:443 [ec2.us-west-2.amazonaws.com/floating_ip_1, ec2.us-west-2.amazonaws.com/floating_ip_2, ec2.us-west-2.amazonaws.com/floating_ip_3] failed: connect timed out
EMS ONTAP Logs:
Wed Mar 18 12:55:21 -0400 [cvo_dr1-01: vifmgr: vifmgr.extRoute.noIpMove:error]: Failed to move LIF. Error: Failed to update external route tables for IP "POST operation failed for URI:http://floating_ip:3080 HTTP code:599 Message:Connect timeout on endpoint URL: \".
Here we can see where the mediator tries to update the route table for one of the data lifs:
In the mediator logs shows we can see the mediator attempting to update the route table for one of the data LIFs:
2026-03-18 16:55:21,716 ERROR: Error returned: [{"route_table": "rtb-0b12345678900", "ip": "10.x.x.x", "message": "Connect timeout on endpoint URL: \"https://ec2.us-west-2.amazonaws.com/\"", "code": "<class 'botocore.exceptions.ConnectTimeoutError'>", "command": "replace_route", "dest": "eni-0000000000
vifmgr.log via ONTAP ASUP
00000003.00192c6f 0e9d7d9e Thu Sep 17 2020 02:27:03 +00:00 [kern_vifmgr:info:8150] [0x808e43600] [FloatingIpMgrImpl::postOperationsToMediator] Failed to update external routing information; failed to send POST msg to mediator, URI: http://10.135.26.11:3080/v1/clusters/2911f4de-1987-11ea-a11d-111111/route_tables/update_all, request: {"IPs":[{"ip":"10.135.26.22","dest":"eni-0614d0e1b8a4a4a4","op":"replace"}]}, status: Timeout was reached, err: Failed to update external route tables for IP "10.135.26.22". The system encountered a timeout waiting for the mediator to complete the operation. Retry the command. If the problem persists, contact technical support personnel for assistance.

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.