Issue mounting Cloud SQL credentials to a container in Kubernetes

10/27/2017

I'm attempting to migrate over to Cloud SQL (Postgres). I have the following deployment in Kubernetes, having followed these instructions https://cloud.google.com/sql/docs/mysql/connect-container-engine :

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: menu-service
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: menu-service
    spec:
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: cloudsql
          emptyDir:
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
      containers:
      - image: gcr.io/cloudsql-docker/gce-proxy:1.11
        name: cloudsql-proxy
        command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                  "-instances=tabb-168314:europe-west2:production=tcp:5432",
                  "-credential_file=/secrets/cloudsql/credentials.json"]
        volumeMounts:
          - name: cloudsql-instance-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
          - name: ssl-certs
            mountPath: /etc/ssl/certs
          - name: cloudsql
            mountPath: /cloudsql
      - name: menu-service
        image: eu.gcr.io/tabb-168314/menu-service:develop
        imagePullPolicy: Always
        env:
        - name: MICRO_BROKER
          value: "nats"
        - name: MICRO_BROKER_ADDRESS
          value: "nats.staging:4222"
        - name: MICRO_REGISTRY
          value: "kubernetes"
        - name: ENV
          value: "staging"
        - name: PORT
          value: "8080"
        - name: POSTGRES_HOST
          value: "127.0.0.1:5432"
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: username
        - name: POSTGRES_PASS
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: password
        - name: POSTGRES_DB
          value: "menus"
        ports:
        - containerPort: 8080

But unfortunately I'm getting this error when trying to update the deployment:

MountVolume.SetUp failed for volume "kubernetes.io/secret/69b0ec99-baaf-11e7-82b8-42010a84010c-cloudsql-instance-credentials" (spec.Name: "cloudsql-instance-credentials") pod "69b0ec99-baaf-11e7-82b8-42010a84010c" (UID: "69b0ec99-baaf-11e7-82b8-42010a84010c") with: secrets "cloudsql-instance-credentials" not found
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "staging"/"menu-service-1982520680-qzwzn". list of unattached/unmounted volumes=[cloudsql-instance-credentials]

Have I missed something here?

-- Ewan Valentine
google-cloud-platform
google-cloud-sql
kubernetes
postgresql

1 Answer

10/27/2017

You are missing (at least) one of the secrets needed to start up this Pod, namely cloudsql-instance-credentials.

From https://cloud.google.com/sql/docs/mysql/connect-container-engine:

You need two secrets to enable your Container Engine application to access the data in your Cloud SQL instance:

  • The cloudsql-instance-credentials secret contains the service account.
  • The cloudsql-db-credentials secret provides the proxy user account and password. (I think you have this created, I can't see an error message about this one)

To create your secrets:

Create the secret containing the Service Account which enables authentication to Cloud SQL:

kubectl create secret generic cloudsql-instance-credentials \
                   --from-file=credentials.json=[PROXY_KEY_FILE_PATH]

[...]

The link above also describes how to create a GCP service account for this purpose, if you don't have one created already.

-- Janos Lenart
Source: StackOverflow