K
Q

Kubernetes app with GCP Cloud SQL does not accept any connections. Full source available on Github

7/12/2018

I've created an example Rails 5 app that uses Google Cloud PostgreSQL. I'm able to run the app locally with docker-compose up, but I'm not able to connect to it remote when I deploy it to GCP. I tried to replicate https://cloud.google.com/ruby/tutorials/bookshelf-on-kubernetes-engine where they use targetPort: http-server

The rails app is published on Github. Am I doing anything obviously wrong? :-|

Running the app locally works

git clone git@github.com:stabenfeldt/k8s-colors.git
docker-compose up -d
docker-compose run colors rake db:create db:migrate
open http://localhost:3000

Create a GKE cluster

gcloud container clusters create color-cluster --num-nodes=2

Setup PostgreSQL Cloud SQL

I followed the instructions from https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine?authuser=1 and updated my config/database.yml and k8s/colors.yml with these values.

Deployed but stuck on ContainerCreating

kubectl apply -f k8s/colors.yml
kubectl get pods

NAME                    READY     STATUS              RESTARTS   AGE
colors-d9f744dc-d5l5v   0/2       ContainerCreating   0          5m
colors-d9f744dc-spmws   0/2       ContainerCreating   0          5m


kubectl logs d9f744dc-d5l5v -c colors  # => Nothing logged

kubectl get deployment

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
colors    2         2         2            0           7m

But fails to connect to the app

kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
colors       LoadBalancer   10.55.245.192   35.228.111.217   80:30746/TCP   1h
kubernetes   ClusterIP      10.55.240.1     <none>           443/TCP        1h

curl 35.228.111.217 # => No response! :-/

kubectl describe svc colors

Name:                     colors
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"colors","namespace":"default"},"spec":{"ports":[{"port":80,"targetPort":3000}]...
Selector:                 app=colors
Type:                     LoadBalancer
IP:                       10.55.252.91
LoadBalancer Ingress:     35.228.203.46
Port:                     <unset>  80/TCP
TargetPort:               3000/TCP
NodePort:                 <unset>  30964/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  Type                  1m    service-controller  ClusterIP -> LoadBalancer
  Normal  EnsuringLoadBalancer  1m    service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   30s   service-controller  Ensured load balancer

k8s/service.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: colors
  labels:
    app: colors

spec:
  replicas: 2
  selector:
    matchLabels:
      app: colors
  template:
    metadata:
      labels:
        app: colors
    spec:
      containers:
        - name: colors
          image: docker.io/stabenfeldt/colors:latest
          ports:
            - name: http-server
              containerPort: 3000


          env:
            - name: POSTGRES_HOST
              value: 127.0.0.1:5432
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password

        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.11
          command: ["/cloud_sql_proxy",
                    "-instances=PROJECT_ID:europe-west1:staging=tcp:5432",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true

      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials

---


apiVersion: v1
kind: Service
metadata:
  name: colors
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 3000
  selector:
    app: colors

kubectl describe deployment

Name:                   colors
Namespace:              default
CreationTimestamp:      Fri, 13 Jul 2018 10:37:06 +0200
Labels:                 app=colors
Annotations:            deployment.kubernetes.io/revision=1
                        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"colors"},"name":"colors","namespace":"default"},"spec":{"repl...
Selector:               app=colors
Replicas:               2 desired | 2 updated | 2 total | 0 available | 2 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=colors
  Containers:
   colors:
    Image:  docker.io/stabenfeldt/colors:latest
    Port:   3000/TCP
    Environment:
      POSTGRES_HOST:      127.0.0.1:5432
      POSTGRES_USER:      <set to the key 'username' in secret 'cloudsql-db-credentials'>  Optional: false
      POSTGRES_PASSWORD:  <set to the key 'password' in secret 'cloudsql-db-credentials'>  Optional: false
    Mounts:               <none>
   cloudsql-proxy:
    Image:  gcr.io/cloudsql-docker/gce-proxy:1.11
    Port:   <none>
    Command:
      /cloud_sql_proxy
      -instances=MY-INSTANCE:europe-west1:staging=tcp:5432
      -credential_file=/secrets/cloudsql/credentials.json
    Environment:  <none>
    Mounts:
      /secrets/cloudsql from cloudsql-instance-credentials (ro)
  Volumes:
   cloudsql-instance-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloudsql-instance-credentials
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   colors-d9f744dc (2/2 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  1m    deployment-controller  Scaled up replica set colors-d9f744dc to 2

kubectl describe service

Name:                     colors
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"colors","namespace":"default"},"spec":{"ports":[{"port":80,"targetPort":3000}]...
Selector:                 app=colors
Type:                     LoadBalancer
IP:                       10.55.252.91
LoadBalancer Ingress:     35.228.203.46
Port:                     <unset>  80/TCP
TargetPort:               3000/TCP
NodePort:                 <unset>  30964/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  Type                  4m    service-controller  ClusterIP -> LoadBalancer
  Normal  EnsuringLoadBalancer  4m    service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   3m    service-controller  Ensured load balancer


Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                10.55.240.1
Port:              https  443/TCP
TargetPort:        443/TCP
Endpoints:         35.228.79.249:443
Session Affinity:  ClientIP
Events:            <none>
-- martins
cloud-sql-proxy
google-cloud-sql
google-kubernetes-engine
kubernetes
ruby-on-rails

Similar Questions

Can't run PyCharm interpreter as root
Docker and Java - FontConfiguration issue
Unhealthy Ingress services
cockroachdb/kubernetes logging lots of errors on stackdriver
Does a kubernetes reconcile have to be quick?
How would I connect to a MySQL on the host machine from inside a docker kubernetes pod?
Prometheus + Kubernetes - do all pods get values even if short lived?

3 Answers

7/18/2018

A working setup can be in my example Rails app at Github.

k8s/colors.yml

# Remember to update MY-INSTANCE

apiVersion: v1
kind: Service
metadata:
  name: colors-frontend
  labels:
    app: colors
    tier: frontend
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: http-server
  selector:
    app: colors
    tier: frontend

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: colors-frontend
  labels:
    app: colors
    tier: frontend
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: colors
        tier: frontend
    spec:
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials

      containers:
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.11
          command: ["/cloud_sql_proxy",
                    "-instances=MY-INSTANCE:europe-west1:development=tcp:5432",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
              - name: cloudsql-instance-credentials
                mountPath: /secrets/cloudsql
                readOnly: true


        - name: colors-app
          image: docker.io/stabenfeldt/colors:1
          imagePullPolicy: Always

          env:
            - name: RAILS_LOG_TO_STDOUT
              value: "true"
            - name: RAILS_ENV
              value: development
            - name: POSTGRES_HOST
              value: 127.0.0.1
            - name: POSTGRES_USERNAME
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password

          ports:
          - name: http-server
            containerPort: 3000
-- martins
Source: StackOverflow
8/22/2018

Your POSTGRES_HOST environment variable needs to be localhost instead of 127.0.0.01:5432. You do not need to add port in the POSTGRES_HOST

-- Narendra
Source: StackOverflow
7/12/2018

I don't see anything wrong outright, but here are a few tips to verifying your Kubernetes Objects look like they should compared to your yamls:

Use the describe command to get more information about objects and make sure they are set up correctly.

For example, if you do kubectl describe deployment <deployment_name> you should verify the following line is present:

Port:       3000/TCP

And for your Service - kubectl describe service <service_name>:

LoadBalancer Ingress:     <PUBLIC_IP>
Port:                     <unset>  80/TCP
TargetPort:               3000/TCP

Finally, I'm not sure if you want to apply the following in your LoadBalancer:

  labels:
    app: colors

Since you are using this label as a selector, it may be doing something funky and trying to load balance to itself instead of your containers with the apps in it.

Also as a side note on your terminology, GCP (Google Cloud Platform) is the overarching name of Google's Services, GKE (Google Kubernetes Engine) is the service providing you with a managed Kuberenetes Cluster.

Hope this helps.

-- kurtisvg
Source: StackOverflow