TL:DR - Sorry if this is basic stuff, I'm learning Kubernetes. I'm trying to create a Kubernetes deployment in IBM Cloud and it's failing. The deployment works fine on my local minikube, but fails in IBM Cloud. Am I required to use the IBM Cloud namespace for deployment or should I be able to pull a Docker image into IBM Cloud from Docker public offerings?
Long version
I'm working through this course on IBM Cloud Kubernetes service (https://courses.cognitiveclass.ai/courses/course-v1:CognitiveClass+CO0401EN+v1/info)
In the prereqs, I need to deploy Redis for a guestbook database.
Those steps are: 1. Clone the repo
$ clone https://github.com/IBM/guestbook.git
cd to /guestbook/v2
Create the Redis controllers and services for both the master and the slave:
$ kubectl create -f redis-master-deployment.yaml
The problem is that when I run:
$ kubectl create -f redis-master-deployment.yaml
The deployment fails:
$ kubectl get deployments -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
redis-master 0/1 1 0 10m redis-master redis:3.2.9 app=redis,role=master
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-master-577bc6fbb-b4trd 0/1 ImagePullBackOff 0 14m 172.30.113.18 10.241.0.11 <none> <none>
I'm guessing here, but it looks like the IBM Cloud Kubernetes service cannot contact Docker to get the image.
$ kubectl describe pods redis-master-577bc6fbb-b4trd
Name: redis-master-577bc6fbb-b4trd
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: 10.241.0.11/10.241.0.11
Start Time: Thu, 05 Mar 2020 12:01:45 -0500
Labels: app=redis
pod-template-hash=577bc6fbb
role=master
Annotations: kubernetes.io/psp: ibm-privileged-psp
Status: Pending
IP: 172.30.113.18
Controlled By: ReplicaSet/redis-master-577bc6fbb
Containers:
redis-master:
Container ID:
Image: redis:3.2.9
Image ID:
Port: 6379/TCP
Host Port: 0/TCP
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rxmp2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-rxmp2:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rxmp2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 600s
node.kubernetes.io/unreachable:NoExecute for 600s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/redis-master-577bc6fbb-b4trd to 10.241.0.11
Warning Failed 13m (x2 over 14m) kubelet, 10.241.0.11 Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 34.197.189.129:443: i/o timeout
Warning Failed 12m kubelet, 10.241.0.11 Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 3.224.75.242:443: i/o timeout
Normal Pulling 11m (x4 over 14m) kubelet, 10.241.0.11 Pulling image "redis:3.2.9"
Warning Failed 11m kubelet, 10.241.0.11 Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 54.210.105.17:443: i/o timeout
Normal BackOff 10m (x6 over 14m) kubelet, 10.241.0.11 Back-off pulling image "redis:3.2.9"
Warning Failed 9m21s (x5 over 14m) kubelet, 10.241.0.11 Error: ErrImagePull
Warning Failed 4m38s (x28 over 14m) kubelet, 10.241.0.11 Error: ImagePullBackOff
I then went to my minikube to test and ran the same command
$ kubectl create -f ./redis-master-deployment.yaml
deployment.apps/redis-master created
$ kubectl get deployments -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
redis-master 1/1 1 1 14s redis-master redis:3.2.9 app=redis,role=master
So I just need a point in the direction of where I'm going wrong here.
Found the issue thanks to @bhpratt
The subnets that the worker nodes are on do not have public IPs. They do have a proxy resource but that has to be enabled on the subnet itself.
IBM Cloud Subnet public gateway pic
That was turned off. Flip the switch and the deployment created successfully.
$ kubectl get deployments -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
redis-master 1/1 1 1 4m50s redis-master redis:3.2.9 app=redis,role=master
Thanks for the additional info. The answer is your worker nodes need public access in order to pull from public docker. Three solutions: