I have 2 GKE cluster both private and public and using cloudproxy as sidecar container for gke app to access cloudsql instance.
public cluster setup for development/testing
Cloud SQL is enabled with both private and public IP. GKE app is using cloudproxy with default option of ip types (public,private) as below Cloud SQL doesn't have any authorized network.
In this case, my app is able to connect CloudSQL and works smoothly. As far as I understand, here connection to cloudsql should be happening with private becuase there is no authorised network configured.
private cluster setup for production
Cloud SQL is enabled with both private and public IP. GKE app is using cloudproxy with default option of ip types (public,private)
cloudsql-proxy setting in deployment file
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy"]
args: ["-instances=$(REAL_DB_HOST)=tcp:$(REAL_DB_PORT)","-credential_file=/secrets/cloudsql/credentials.json"]case 1 Cloud SQL doesn't have any authorized network. Result: Application is not able to connect with Cloud SQL
case 2 Cloud SQL have private GKE NAT gateway as authorized network Result: Application is not able to connect with Cloud SQL
May be removing cloudproxy from application will work (I am yet to test) but it discourages the usage of proxy during dev env as it will need changes in deployment file during production deployment.
I am not able to understand what is causing the connection failure with cloudproxy in gke private cluster. Should we not use cloudproxy in private cluster?
Update The reason due to which cloud proxy not able to connect cloud sql was disabled Cloud SQL Admin API. I have updated my answer in answer section.
@kurtisvg has provided an informative answer to it.
However the real issue was SQL Admin API and enabling it fixed the issue. After looking into the logs I found below entry.
Error 403: Access Not Configured. Cloud SQL Admin API has not been used in project XXXXXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?
It looks like the question here is "Should we use the Cloud SQL proxy in a private cluster?" and that answer is "it depends". It's not required to connect, but it allows for more security because you can restrict unnecessary access to your Cloud SQL server.
The Cloud SQL proxy doesn't provide connectivity for you application - it only provides authentication. It has to be able to connect via the existing path, but then uses the Service Account's IAM roles to authenticate the connection. This also means that it doesn't have to come from a whitelisted network because it's been authenticated by a different means.
If you want to use the proxy to connect via Private IP (instead of defaulting to public), use the -ip_address_types=PRIVATE - this will tell the proxy to connect with the instance's Private IP instead. (Please note that if the proxy lacks a network path (eg, isn't on the VPC) that proxy will still be unable to connect.)