How can I get kubernetes resources to succesfully connect to an RDS instance in another vpc

12/11/2018

Background: I have a kubernetes cluster set up in one AWS account that needs to access data in an RDS MySQL instance in a different account and I can't seem to get the settings correct to allow traffic to flow.

What I've tried so far:

  • Setup a peering connection between the two VPCs. They are in the same region, us-east-1.
  • Created Route table entries in each account to point traffic on the corresponding subnet to the peering connection.
  • Created a security group in the RDS VPC to allow traffic from the kubernetes subnets to access MySql.
  • Made sure DNS Resolution is enabled on both VPC's.

Kubernetes VPC details (Requester)

This contains 3 EC2's (looks like each has its own subnet) that house my kubernetes cluster. I used EKS to set this up.

The route table rules I set up have the 3 subnets associated, and point the RDS VPC CIDR block at the peering connection.

RDS VPC details (Accepter)

This VPC contains the mysql RDS instance, as well as some other resources. The RDS instance has quite a few VPC security groups assigned to it for access from our office IP's etc. It has Public Accessibility set to true.

I repeated the route table setup (in reverse) and pointed back to the K8s VPC subnet / peering connection.

Testing

To test the connection, I've tried 2 different ways. The application that needs to access mysql is written in node, so I just wrote a test connector and example query and it times out.

I also tried netcat from a terminal in the pod running in the kubernetes cluster.

nc -v {{myclustername}}.us-east-1.rds.amazonaws.com 3306

Which also times out. It seems to be trying to hit the correct mysql instance IP though so I'm not sure if that means my routing rules are working right from the k8s vpc side.

DNS fwd/rev mismatch: ec2-XXX.compute-1.amazonaws.com != ip-{{IP OF MY MYSQL}}.ec2.internal

I'm not sure what steps to take next. Any direction would be greatly appreciated.

Side Note: I've read thru this Kubernetes container connection to RDS instance in separate VPC I think I understand what's going on there. My CIDR blocks do not conflict with the default K8s ips (10.0...) so my problem seems to be different.

-- Matt H
amazon-web-services
kubernetes
vpc

1 Answer

3/28/2020

I know this was asked a long time ago, but I just ran into this problem as well.

It turns out I was editing the wrong AWS routing table! When I ran kops to create my cluster, it created a new VPC with its own routing table but also another routing table! I needed to add the peer connection route to the cluster's routing table instead of the VPC's Main routing table.

Routing tables

-- Foo L
Source: StackOverflow