kubeadm init change imageRepository

10/15/2018

I'm trying to start a kubernetes cluster but with a different url for kubernetes to pull it's images. AFAIK, it's only possible through config file.

I'm not familiar with the config file, so I started with a simple one:

apiVersion: kubeadm.k8s.io/v1alpha2
imageRepository: my.internal.repo:8082
kind: MasterConfiguration
kubernetesVersion: v1.11.3

And ran the command kubeadm init --config file.yaml After some time, it fails with the following error:

[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
I1015 12:05:54.066140   27275 kernel_validator.go:81] Validating kernel version
I1015 12:05:54.066324   27275 kernel_validator.go:96] Validating kernel config
        [WARNING Hostname]: hostname "kube-master-0" could not be reached
        [WARNING Hostname]: hostname "kube-master-0" lookup kube-master-0 on 10.11.12.246:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [kube-master-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.10.5.189]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [kube-master-0 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [kube-master-0 localhost] and IPs [10.10.5.189 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

                Unfortunately, an error has occurred:
                        timed out waiting for the condition

                This error is likely caused by:
                        - The kubelet is not running
                        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
                        - No internet connection is available so the kubelet cannot pull or find the following control plane images:
                                - my.internal.repo:8082/kube-apiserver-amd64:v1.11.3
                                - my.internal.repo:8082/kube-controller-manager-amd64:v1.11.3
                                - my.internal.repo:8082/kube-scheduler-amd64:v1.11.3
                                - my.internal.repo:8082/etcd-amd64:3.2.18
                                - You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
                                  are downloaded locally and cached.

                If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                        - 'systemctl status kubelet'
                        - 'journalctl -xeu kubelet'

                Additionally, a control plane component may have crashed or exited when started by the container runtime.
                To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
                Here is one example how you may list all Kubernetes containers running in docker:
                        - 'docker ps -a | grep kube | grep -v pause'
                        Once you have found the failing container, you can inspect its logs with:
                        - 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

I checked kubelet status with systemctl status kubelet, and it's running.

I sucessfully tried to manully pull the images with:

docker pull my.internal.repo:8082/kubee-apiserver-amd64:v1.11.3

However, 'docker ps -a returns' no containers.

The journalctl -xeu kubelet show lots of connection refused and get requests to k8s.io that I'm struggling to understand the root error.

Any ideas?

Thanks in advance!

Edit 1: I tried to manually open the ports, but nothing changed. [centos@kube-master-0 ~]$ sudo firewall-cmd --zone=public --list-ports 6443/tcp 5000/tcp 2379-2380/tcp 10250-10252/tcp

I also changed the kube version from 1.11.3 to 1.12.1, but nothing changed.

Edit 2: I realized that kubelet is trying to pull from k8s.io repo, which means I changed kubeadm internal repo only. I need to do the same with kubelet.

Oct 22 11:10:06 kube-master-1-120 kubelet[24795]: E1022 11:10:06.108764   24795 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to...on refused
Oct 22 11:10:06 kube-master-1-120 kubelet[24795]: E1022 11:10:06.110539   24795 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v...on refused

Any ideas?

-- fsakiyama
kubeadm
kubernetes

2 Answers

10/22/2018

you solved half of the problem, may the final solution is to edit the kubelet (/etc/systemd/system/kubelet.service.d/10-kubeadm.conf) init file. You need to set --pod_infra_container_image parameter that will reference to the pause container image that was pulled through your internal repository. The image name will be like this: my.internal.repo:8082/pause:[version].

The reason is that the kubelet can't get the new image tag to refers it.

-- Everson Leal
Source: StackOverflow

10/15/2018

I will post my comment as an answer due to formatting text unavailability in proper way using comment:

What happen if you will try to download images before cluster init? Example:

master-config.yaml:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.3

Command:

root@kube-master-01:~# kubeadm config images pull --config="/root/master-config.yaml"

Output:

[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.11.3
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.11.3
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.11.3
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.11.3
[config/images] Pulled k8s.gcr.io/pause:3.1
[config/images] Pulled k8s.gcr.io/etcd:3.2.18
[config/images] Pulled k8s.gcr.io/coredns:1.2.2

P.S: add imageRepository: my.internal.repo:8082 before trying.

-- VKR
Source: StackOverflow