Connecting to an AWS database from GKE

11/14/2018

I am having troubles connecting a Google Cloud Platform Kubernetes pod to an external MySQL running on AWS.

Here's my deployment file (some sensitive parts replaced by ***):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: watches-v1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: watches-v1
  template:
    metadata:
      labels:
        app: watches-v1
    spec:
      containers:
      - name: watches-v1
        image: silasberger/watches:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 3000
        env:
        - name: MYSQL_HOST
          value: "***.eu-west-1.rds.amazonaws.com"
        - name: MYSQL_DB
          value: "***"
        - name: MYSQL_USER
          value: "***"
        - name: MYSQL_PASS
          value: "***"
        - name: API_USER
          value: "***"
        - name: API_PASS
          value: "***"

This is the Dockerfile which I build and push to Dockerhub as watches:1.0:

FROM node:8

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 3000

ENV MICROSERVICE="watches"
ENV WATCHES_API_VERSION="1"

CMD [ "npm", "start" ]

The following things work:

  • Connect to the AWS MySQL instance from a bash, using the mysql command
  • running the Docker image in a local container, no errors, everything as expected

However, as soon as I apply the deployment in my Kubernetes cluster, the pods aren't able to connect to the AWS DB. The application starts, I can access the swagger page, but when I run the kubectl logs <pod-name> command, I always get this error:

Unable to connect to the database: { SequelizeConnectionError: connect ETIMEDOUT
    at Utils.Promise.tap.then.catch.err (/usr/src/app/node_modules/sequelize/lib/dialects/mysql/connection-manager.js:149:19)
    at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
    at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
    at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
    at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
    at Immediate.Async.drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
  name: 'SequelizeConnectionError',
  parent: 
   { Error: connect ETIMEDOUT
    at Connection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:192:13)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect',
     fatal: true },
  original: 
   { Error: connect ETIMEDOUT
    at Connection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:192:13)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect',
     fatal: true } }

It chooses the correct host, DB name and credentials (as indicated by a previous part of the log not shown here), but it apparently can't connect to it. As you can see, the application is written in Node.js and uses Sequelize.

All the research I have done so far pointed to a firewall issue, so I set the following VPC rule on the Google Cloud Platform for that project:

$ gcloud compute firewall-rules describe allow-all-outbound
allowed:
- IPProtocol: all
creationTimestamp: '2018-11-14T02:51:20.808-08:00'
description: Allow all inbound connections
destinationRanges:
- 0.0.0.0/0
direction: EGRESS
disabled: false
id: '7178441953737326791'
kind: compute#firewall
name: allow-mysql-outbound
network: https://www.googleapis.com/compute/v1/projects/adept-vine-222109/global/networks/default
priority: 1000
selfLink: https://www.googleapis.com/compute/v1/projects/adept-vine-222109/global/firewalls/allow-mysql-outbound

Since this didn't change anything, I also tried adding the same rule again, with direction INGRESS, but that didn't work either (as I expected).

I am totally new to the Google Cloud Platform and to Kubernetes, so maybe this is just a dumb mistake, but I'm really out of ideas on how to get it to work.

-- Silas Berger
amazon-web-services
google-cloud-platform
kubernetes
mysql
sequelize.js

1 Answer

11/14/2018

As it turns out, the problem was on the AWS side. Thanks Jacob Tomlinson for the suggestion.

While Public Accessibility was activated for the AWS MySQL instance, it apparently didn't allow access from all sources. I'm not sure why it worked from my local machine, but anyway.

I was able to solve it by adding a security group in AWS that allows inbound traffic on all ports and with all protocols for the source 0.0.0.0/0. I then associated this security group with my MySQL instance (go to the instance, click Modify, go to Network & Security settings, choose the newly created group, save changes). I will still need to tweak this rule from a security perspective, but at least it all works now.

-- Silas Berger
Source: StackOverflow