I am trying to run a docker image from private GCR using KubernetesPodOperator in Cloud Composer, but getting the following error:

ERROR: Pod launching failed : Pod took too long to start

I have tried the following till now:

1. At first I tried increasing the "startup_timeout_seconds" but it didn't help.

2. Looking at the Composer created GKE cluster logs gave me the following error:

   Failed to apply default image tag "docker pull us.gcr.io/my-proj-name/myimage- name:latest": couldn't parse image reference "docker pull us.gcr.io/my-proj- name/myimage-name:latest": invalid reference format: InvalidImageName

I tried pulling the same docker image on my local machine from my private GCR and it worked fine, not sure where is the issue.

3. This link https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod tells me that

   "All pods in a cluster will have read access to images in this registry. The kubelet will authenticate to GCR using the instance’s Google service account. The service account on the instance will have a https://www.googleapis.com/auth/devstorage.read_only, so it can pull from the project’s GCR, but not push"

   which means the pod should be able to pull image from GCR. FYI, I am using a service account to provision my composer env and it has sufficient permission to read from GCS bucket.

   Also, I did the following steps to add secret :

   gcloud container clusters get-credentials <cluster_name>

   kubectl create secret generic gc-storage-rw-key --from-file=key.json=<path_to_serv_accnt_key>

   secret_file = secret.Secret(
       deploy_type='volume',
       deploy_target='/tmp/secrets/google',
       secret='gc-storage-rw-key',
       key='<path of serv acct key file>.json')

   Refer it as secrets=[secret_file] inside KubernetesPodOperator operator in DAG

4. I have added image_pull_policy='Always' in my DAG as well but not working...

5. For reference: my CircleCI config.yml contains following

- run: echo ${GOOGLE_AUTH} > ${HOME}/gcp-key.json
- run: docker build --rm=false -t us.gcr.io/${GCP_PROJECT}/${IMAGE_NAME}:latest .
- run: gcloud auth activate-service-account --key-file ${HOME}/gcp-key.json
- run: gcloud --quiet config set project ${GCP_PROJECT}
- run: gcloud docker -- push us.gcr.io/${GCP_PROJECT}/${IMAGE_NAME}:latest

Could anyone please guide me?