We launched a Cloud Composer cluster and want to use it to move data from Cloud SQL (Postgres) to BQ. I followed the notes about doing this mentioned at these two resources:

Google Cloud Composer and Google Cloud SQL

https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine

We launch a pod running the cloud_sql_proxy and launch a service to expose the pod. The problem is that Cloud Composer cannot see the service stating the error when attempting to use an ad-hoc query to test:

cloud not translate host name "sqlproxy-service" to address: Name or service not known"

Trying by the service IP address results in the page timing out.

The -instances passed to cloud_sql_proxy work when used in a local environment or cloud shell. The log files seem to indicate no connection is ever attempted

me@cloudshell:~ (my-proj)$ kubectl logs -l app=sqlproxy-service
me@2018/11/15 13:32:59 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.

    2018/11/15 13:32:59 using credential file for authentication; email=my-service-account@service.iam.gserviceaccount.com
    2018/11/15 13:32:59 Listening on 0.0.0.0:5432 for my-proj:my-ds:my-db
    2018/11/15 13:32:59 Ready for new connections

I see a comment here https://stackoverflow.com/a/53307344/1181412 that possibly this isn't even supported?

Airflow

YAML

apiVersion: v1
kind: Service
metadata:
  name: sqlproxy-service
  namespace: default
  labels:
    app: sqlproxy
spec:
  ports:
  - port: 5432
    protocol: TCP
    targetPort: 5432
  selector:
    app: sqlproxy
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sqlproxy
  labels:
    app: sqlproxy
spec:
  selector:
    matchLabels:
      app: sqlproxy
  template:
    metadata:
      labels:
        app: sqlproxy
    spec:
      containers:
        - name: cloudsql-proxy
          ports:
          - containerPort: 5432
            protocol: TCP
          image: gcr.io/cloudsql-docker/gce-proxy:latest
          imagePullPolicy: Always
          command: ["/cloud_sql_proxy",
                    "-instances=my-proj:my-region:my-db=tcp:0.0.0.0:5432",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          securityContext:
            runAsUser: 2  # non-root user
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials