Pulling images from private google container registry with kubeflow on minikube
We are having trouble giving a container within a pipeline uploaded to Kubeflow access to a private custom docker image stored in a google container registry. We are running kubeflow on top of a kubernetes cluster run on minikube. Can someone help us understand how to add the access token/service account to the Kubeflow deployment? We have read a couple of docs that achieve this on a custom Kubernetes deployment but not on a Kubeflow deployment.
The error we get when running the pipeline on Kubeflow is: This step is in Pending state with this message: ImagePullBackOff: Back-off pulling image
This is the pipeline code that calls the image. 
Thank you!!
Similar Questions
1 Answer
This is issues can occur in some scenarios like:
Your kubeflow setup (Kubernetes cluster) and GCR are in different project
No GCR secret for the ml-pipeline service account which is responsible to run the pipeline. (you can see this kubectl --namespace=kubeflow get serviceaccount)
In your case, I think it is the second scenario. Though the following path will work on both scenarios.
- Create service_account.json with sufficient permission (GCR needs storage permission so give 'Storage admin') using the GCP console
Select “API & Services” > “Credentials”Select “Create credentials” > “Services Account Key” > “Create New Services Account”- Add a Kubernetes Secret in Kubernetes Cluster to access GCR
kubectl create secret docker-registry $SECRETNAME \
--docker-server=https://gcr.io \
--docker-username=_json_key \
--docker-email=user@example.com \
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key- Above method is for default service account. But patch this in Kufelow namespace
kubectl --namespace=kubeflow create secret docker-registry $SECRETNAME \
--docker-server=https://gcr.io \
--docker-username=_json_key \
--docker-email=user@example.com \
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key- Patching GCR secret with respective service account
# For Kubeflow specific problem path pipeline-runner serviceaccount
kubectl --namespace=kubeflow patch serviceaccount pipeline-runner -p '{"imagePullSecrets": [{"name": "$SECRETNAME"}]}'