cloudsql-proxy on GKE returns "dial tcp: lookup 127.0.0.1:3306: no such host" with a golan app

1/23/2020

I'm trying to deploy a Go service that uses a Postgres connection on GKE.

The deployment manifest looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: storage-service
  labels:
    app: storage-service
spec:
  selector:
    matchLabels:
      app: storage-service
  template:
    metadata:
      labels:
        app: storage-service
    spec:
      containers:
        - name: app
          image: my-image
          ports:
            - containerPort: 80
          # The following environment variables will contain the database host,
          # user and password to connect to the PostgreSQL instance.
          env:
            - name: SERVICE_DATABASE_NAME
              value: db_storage
            - name: DATABASE_MIGRATIONS_DIR
              value: ./migrations
            - name: MAX_FILE_SIZE
              value: "20000000"
            - name: SERVICE_BASE_PATH
              value: storage
            - name: SERVICE_VERSION
              value: v1
            - name: SERVICE_DESCRIPTION
              value: "A service to store and retrieve files"
            - name: SERVICE_NAME
              value: storage-service
            - name: DATABASE_HOST
              value: 127.0.0.1:3306
            # [START cloudsql_secrets]
            - name: ENV
              valueFrom:
                secretKeyRef:
                  name: environment
                  key: ENV
            - name: PORT
              valueFrom:
                secretKeyRef:
                  name: environment
                  key: DEFAULT_APPLICATION_PORT
            # - name: DATABASE_HOST
            #   valueFrom:
            #     secretKeyRef:
            #       name: sql-proxy
            #       key: DB_HOST
            - name: DATABASE_USER
              valueFrom:
                secretKeyRef:
                  name: sql-proxy
                  key: DB_USER
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: sql-proxy
                  key: DB_PASS
            - name: DATABASE_SSL_MODE
              valueFrom:
                secretKeyRef:
                  name: sql-proxy
                  key: DB_SSL_MODE
            - name: GCLOUD_PUBLIC_STORAGE_BUCKET
              valueFrom:
                secretKeyRef:
                  name: storage-buckets
                  key: STORAGE_PUBLIC_BUCKET_NAME
            - name: GCLOUD_PRIVATE_STORAGE_BUCKET
              valueFrom:
                secretKeyRef:
                  name: storage-buckets
                  key: STORAGE_PRIVATE_BUCKET_NAME
            # [END cloudsql_secrets]
          volumeMounts:
          - name: firebase-credentials-volume
            mountPath: /storage_service
            readOnly: true
        # Change <INSTANCE_CONNECTION_NAME> here to include your GCP
        # project, the region of your Cloud SQL instance and the name
        # of your Cloud SQL instance. The format is
        # $PROJECT:$REGION:$INSTANCE
        # [START proxy_container]
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.14
          command: ["/cloud_sql_proxy",
                    "-instances=${DB_INSTANCE_CONNECTION_NAME}=tcp:3306",
                    # If running on a VPC, the Cloud SQL proxy can connect via Private IP. See:
                    # https://cloud.google.com/sql/docs/mysql/private-ip for more info.
                    # "-ip_address_types=PRIVATE",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          # [START cloudsql_security_context]
          securityContext:
            runAsUser: 2  # non-root user
            allowPrivilegeEscalation: false
          # [END cloudsql_security_context]
          env:
            - name: DB_INSTANCE_CONNECTION_NAME
              valueFrom:
                secretKeyRef:
                  name: sql-instance
                  key: DB_INSTANCE_CONNECTION_NAME
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
        # [END proxy_container]
      # [START volumes]
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: sql-instance
        - name: firebase-credentials-volume
          secret:
            secretName: firebase
            items:
            - key: firebase_reader.json
              path: firebase.json
      # [END volumes]

When I deploy it to my GKE, we have this logs:

  • From sql-proxy container:
> current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
> using credential file for authentication; email=cloudsql-proxy-reader@corp.iam.gserviceaccount.com
> Listening on 127.0.0.1:3306 for mycorp:europe-west4:ds-db-dev-test05
> Ready for new connections
  • App container:
> Dial tcp: lookup 127.0.0.1:3306: no such host

I also put a log to see my postgres connection string:

user=postgres password=PaSS sslmode=disable host=127.0.0.1:3306

And the fact is that my service can't connect with the sql-proxy. Can someone help me?

-- Javier Coronel
gcloud
google-kubernetes-engine

2 Answers

1/27/2020

Thanks for your answers.

The final problem was a mixture of everything. The GO's Postgres driver does not expect a host with a port (like localhost:5432), and that was the log that was printed.

I changed the port to the default Postgres as Mahboob said and left the host on only localhost as Ulrich Eckhardt comments and everything works fine.

Thanks to all of you

-- Javier Coronel
Source: StackOverflow

1/23/2020

According to GCP documentation, please follow the article Troubleshooting Cloud SQL Proxy and click here to troubleshoot cloud-proxy connections issue.

However, the default TCP port for PostgreSQL is usually 5432. Here, on your manifest you used 3306 port which is default port of MySQL. Thus, change the port 3306 to 5432 and then try again.

I would like to suggest you check your firewall rules for port 3307 as this port is used by the Cloud SQL Proxy. If you have a firewall on your network that might blocks port 3307. To use Cloud SQL proxy, authorize this port for outbound connections. In addition to this, please use proper IAM role for the service account e.i use "Cloud SQL Viewer" role, try to connect. I hope this will help you to solve the connection problem.

-- Mahboob
Source: StackOverflow