I have a local minikube setup using hyperkit VM. I installed helm on my system and the client was installed successfully but tiller is not installing in the cluster. The status for the pod is ImagePullBackOff
I tried installing different versions of the image with the --tiller-image flag but it still fails. If I do a normal docker pull of the same image in my system it works.
The command helm version gives this result:
Client: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
Error: could not find a ready tiller podThe command kubectl -n kube-system get pods gives this results:
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-bh6w7 1/1 Running 0 146m
coredns-fb8b8dccf-gskc8 1/1 Running 1 146m
etcd-minikube 1/1 Running 0 145m
kube-addon-manager-minikube 1/1 Running 0 145m
kube-apiserver-minikube 1/1 Running 0 145m
kube-controller-manager-minikube 1/1 Running 0 145m
kube-proxy-jqb9b 1/1 Running 0 146m
kube-scheduler-minikube 1/1 Running 0 145m
storage-provisioner 1/1 Running 0 146m
tiller-deploy-548df79d66-xsngk 0/1 ImagePullBackOff 0 27mThe command kubectl -n kube-system describe pod tiller-deploy-548df79d66-xsngk fives the below result:
Name: tiller-deploy-548df79d66-xsngk
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: minikube/192.168.64.56
Start Time: Sat, 06 Jul 2019 22:41:54 +0530
Labels: app=helm
name=tiller
pod-template-hash=548df79d66
Annotations: <none>
Status: Pending
IP: 172.17.0.5
Controlled By: ReplicaSet/tiller-deploy-548df79d66
Containers:
tiller:
Container ID:
Image: gcr.io/kubernetes-helm/tiller:v2.12.1
Image ID:
Ports: 44134/TCP, 44135/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Liveness: http-get http://:44135/liveness delay=1s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:44135/readiness delay=1s timeout=1s period=10s #success=1 #failure=3
Environment:
TILLER_NAMESPACE: kube-system
TILLER_HISTORY_MAX: 0
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-6w54d (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-6w54d:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6w54d
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 21h default-scheduler Successfully assigned kube-system/tiller-deploy-548df79d66-xsngk to minikube
Warning Failed 21h kubelet, minikube Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:46818->192.168.64.1:53: read: connection refused
Warning Failed 21h kubelet, minikube Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:32929->192.168.64.1:53: read: connection refused
Warning Failed 21h kubelet, minikube Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:59446->192.168.64.1:53: read: connection refused
Warning Failed 21h (x4 over 21h) kubelet, minikube Error: ErrImagePull
Normal Pulling 21h (x4 over 21h) kubelet, minikube Pulling image "gcr.io/kubernetes-helm/tiller:v2.12.1"
Warning Failed 21h kubelet, minikube Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:58643->192.168.64.1:53: read: connection refused
Warning Failed 21h (x7 over 21h) kubelet, minikube Error: ImagePullBackOff
Normal BackOff 21h (x110 over 21h) kubelet, minikube Back-off pulling image "gcr.io/kubernetes-helm/tiller:v2.12.1"I am not behind any proxy if that helps.
Update 1:
I ssh-ed into minikube and tried to pull the docker image manually but I got the error:
Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:46912->192.168.64.1:53: read: connection refusedUpdate 2:
This issue is not only for just helm, but for any image. I usw Quay.io for images and the cluster can't reach that also, I tried kafka and that fails too. If I ssh into minikube, update resolv and then manually do a docker pull inside minikube, the pod spins up.
Reuse the docker daemon for minikube with the command,
eval $(minikube docker-env)and then pull the tiller image like this,
docker pull gcr.io/kubernetes-helm/tiller:v2.12.1Now minikube will have the image downloaded. Delete the existing tiller pod with this command,
kubectl delete pod tiller-deploy-548df79d66-xsngk -n kube-systemThe explanation is this. Since it looks like the ImagePullPolicy is not mentioned in the tiller deployment and also as the default ImagePullPolicy is IfNotPresent, minikube might not download the image and will try to use the already dowloaded image.
Update: The reason that helm was not able to install tiller, being that the resolv.conf file of minikube container had nameserver 192.168.64.1 from the previous network that he got connected to while creating the cluster. So minikube was not able to pull tiller docker image. It might probably be due to the fact that docker sets up the nameserver based on the network you are connected to.
This link may help.