kubeadm init fails with : x509: certificate signed by unknown authority

4/22/2019

Following https://kubernetes.io/blog/2019/03/15/kubernetes-setup-using-ansible-and-vagrant/ , tried to setup Kubernetes using vagrant on Mac. With the Ansible Playbook Step:

 - name: Initialize the Kubernetes cluster using kubeadm
    command: kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10"  --node-name k8s-master --pod-network-cidr=192.168.0.0/16

I get error:

fatal: [k8s-master]: FAILED! => {"changed": true, "cmd": ["kubeadm", "init", "--apiserver-advertise-address=192.168.50.10", "--apiserver-cert-extra-sans=192.168.50.10", "--node-name", "k8s-master", "--pod-network-cidr=192.168.0.0/16"], "delta": "0:00:03.446240", "end": "2019-04-22 08:32:03.655520", "msg": "non-zero return code", "rc": 1, "start": "2019-04-22 08:32:00.209280", "stderr": "I0422 08:32:00.877733 5038 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \"https://dl.k8s.io/release/stable-1.txt\": Get https://dl.k8s.io/release/stable-1.txt: x509: certificate signed by unknown authority\nI0422 08:32:00.877767 5038 version.go:97] falling back to the local client version: v1.14.1\n\t[WARNING IsDockerSystemdCheck]: detected \"cgroupfs\" as the Docker cgroup driver. The recommended driver is \"systemd\". Please follow the guide at https://kubernetes.io/docs/setup/cri/\nerror execution phase preflight: [preflight] Some fatal errors occurred:\n\t[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority\n,

so I tried to run the Kubeadm init command manually:

kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10"  --node-name k8s-master --pod-network-cidr=192.168.0.0/16  --ignore-preflight-errors all
I0422 08:51:06.815553    6537 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: x509: certificate signed by unknown authority
I0422 08:51:06.815587    6537 version.go:97] falling back to the local client version: v1.14.1

I tried the same command with --ignore-preflight-errors all

kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10"  --node-name k8s-master --pod-network-cidr=192.168.0.0/16  --ignore-preflight-errors all
I0422 08:51:35.741958    6809 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: x509: certificate signed by unknown authority
I0422 08:51:35.742030    6809 version.go:97] falling back to the local client version: v1.14.1
[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
    [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
    [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
    [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
    [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
    [WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
    [WARNING ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.10: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
    [WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.50.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.50.10 127.0.0.1 ::1]
---
- hosts: all
  become: true
  tasks:
  - name: Install packages that allow apt to be used over HTTPS
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - apt-transport-https
      - ca-certificates
      - curl
      - gnupg-agent
      - software-properties-common

  - name: Add an apt signing key for Docker
    apt_key:
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present

  - name: Add apt repository for stable version
    apt_repository:
      repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
      state: present

  - name: Install docker and its dependecies
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - docker-ce
      - docker-ce-cli
      - containerd.io
    notify:
      - docker status

  - name: Add vagrant user to docker group
    user:
      name: vagrant
      group: docker
/Initialize
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.50.10 192.168.50.10]
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
    timed out waiting for the condition

This error is likely caused by:
    - The kubelet is not running
    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
    - 'systemctl status kubelet'
    - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
    - 'docker ps -a | grep kube | grep -v pause'
    Once you have found the failing container, you can inspect its logs with:
    - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

based on the valuable suggestions, I tried the command:

kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10" --pod-network-cidr=192.168.0.0/16  --kubernetes-version="v1.14.1" --ignore-preflight-errors all --cert-dir=/etc/ssl/cert

but got error response:

[init] Using Kubernetes version: v1.14.1 [preflight] Running pre-flight checks [WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists [WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists [WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists [WARNING FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Port-10250]: Port 10250 is in use [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority , error: exit status 1 [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority , error: exit status 1 [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority , error: exit status 1 [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority , error: exit status 1 [WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority , error: exit status 1 [WARNING ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.10: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority , error: exit status 1 [WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority , error: exit status 1 [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/ssl/cert" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.50.10 192.168.50.10] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.50.10 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.50.10 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" error execution phase kubeconfig/admin: a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert

command:

kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10" --pod-network-cidr=192.168.0.0/16  --kubernetes-version="v1.14.1" --ignore-preflight-errors all --cert-dir=/etc/kubernetes/pki

error trace:

Unfortunately, an error has occurred:
    timed out waiting for the condition

This error is likely caused by:
    - The kubelet is not running
    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
    - 'systemctl status kubelet'
    - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
    - 'docker ps -a | grep kube | grep -v pause'
    Once you have found the failing container, you can inspect its logs with:
    - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

furthermore:

root@k8s-master:~# systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2019-04-24 00:13:07 UTC; 9min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 9746 (kubelet)
    Tasks: 16
   Memory: 27.7M
      CPU: 9.026s
   CGroup: /system.slice/kubelet.service
           └─9746 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1

Apr 24 00:22:19 k8s-master kubelet[9746]: E0424 00:22:19.652197    9746 kubelet.go:2244] node "k8s-master" not found
Apr 24 00:22:19 k8s-master kubelet[9746]: E0424 00:22:19.711938    9746 controller.go:115] failed to ensure node lease exists, will retry in 7s, error: Get https://192.168.50.10:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/k8s-master?timeout=10s: dial tcp 192.168.50.10:6443: connect: connection refused
Apr 24 00:22:19 k8s-master kubelet[9746]: E0424 00:22:19.752613    9746 kubelet.go:2244] node "k8s-master" not found
Apr 24 00:22:19 k8s-master kubelet[9746]: E0424 00:22:19.818002    9746 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://192.168.50.10:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.50.10:6443: connect: connection refused
Apr 24 00:22:19 k8s-master kubelet[9746]: E0424 00:22:19.859028    9746 kubelet.go:2244] node "k8s-master" not found
Apr 24 00:22:19 k8s-master kubelet[9746]: E0424 00:22:19.960182    9746 kubelet.go:2244] node "k8s-master" not found
Apr 24 00:22:20 k8s-master kubelet[9746]: E0424 00:22:20.018188    9746 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://192.168.50.10:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.50.10:6443: connect: connection refused
Apr 24 00:22:20 k8s-master kubelet[9746]: E0424 00:22:20.061118    9746 kubelet.go:2244] node "k8s-master" not found
Apr 24 00:22:20 k8s-master kubelet[9746]: E0424 00:22:20.169412    9746 kubelet.go:2244] node "k8s-master" not found
Apr 24 00:22:20 k8s-master kubelet[9746]: E0424 00:22:20.250762    9746 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://192.168.50.10:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.50.10:6443: connect: connection refused
root@k8s-master:~#

looking at all docker containers:

root@k8s-master:~# docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                       PORTS               NAMES
a22812e3c702        20a2d7035165                         "/usr/local/bin/kube…"   4 minutes ago       Up 4 minutes                                     k8s_kube-proxy_kube-proxy-t7nq9_kube-system_20f8d57d-6628-11e9-b099-080027ee87c4_0
b2a89f8418bb        k8s.gcr.io/pause:3.1                 "/pause"                 4 minutes ago       Up 4 minutes                                     k8s_POD_kube-proxy-t7nq9_kube-system_20f8d57d-6628-11e9-b099-080027ee87c4_0
6c327b9d36f2        cfaa4ad74c37                         "kube-apiserver --ad…"   5 minutes ago       Up 5 minutes                                     k8s_kube-apiserver_kube-apiserver-k8s-master_kube-system_0260f2060ab76fc71c634c4499054fe6_1
a1f1b3396810        k8s.gcr.io/etcd                      "etcd --advertise-cl…"   5 minutes ago       Up 5 minutes                                     k8s_etcd_etcd-k8s-master_kube-system_64388d0f4801f9b4aa01c8b7505258c9_0
0a3619df6a61        k8s.gcr.io/kube-controller-manager   "kube-controller-man…"   5 minutes ago       Up 5 minutes                                     k8s_kube-controller-manager_kube-controller-manager-k8s-master_kube-system_07bbd1f39b3ac969cc18015bbdce8871_0
ffb435b6adfe        k8s.gcr.io/kube-apiserver            "kube-apiserver --ad…"   5 minutes ago       Exited (255) 5 minutes ago                       k8s_kube-apiserver_kube-apiserver-k8s-master_kube-system_0260f2060ab76fc71c634c4499054fe6_0
ffb463d4cbc6        k8s.gcr.io/pause:3.1                 "/pause"                 5 minutes ago       Up 5 minutes                                     k8s_POD_etcd-k8s-master_kube-system_64388d0f4801f9b4aa01c8b7505258c9_0
a9672f233952        k8s.gcr.io/kube-scheduler            "kube-scheduler --bi…"   5 minutes ago       Up 5 minutes                                     k8s_kube-scheduler_kube-scheduler-k8s-master_kube-system_f44110a0ca540009109bfc32a7eb0baa_0
2bc0ab68870b        k8s.gcr.io/pause:3.1                 "/pause"                 5 minutes ago       Up 5 minutes                                     k8s_POD_kube-controller-manager-k8s-master_kube-system_07bbd1f39b3ac969cc18015bbdce8871_0
667ae6988f2b        k8s.gcr.io/pause:3.1                 "/pause"                 5 minutes ago       Up 5 minutes                                     k8s_POD_kube-apiserver-k8s-master_kube-system_0260f2060ab76fc71c634c4499054fe6_0
b4e6c37f5300        k8s.gcr.io/pause:3.1                 "/pause"                 5 minutes ago       Up 5 minutes                                     k8s_POD_kube-scheduler-k8s-master_kube-system_f44110a0ca540009109bfc32a7eb0baa_0
-- kamal
ansible
kubernetes
vagrant

3 Answers

4/25/2019

so as far as "kubeadm init fails with : x509: certificate signed by unknown authority" although I really appreciate all the valuable help which helped a lot, the x509 cert issue was resolved by the following in the Ansible playbook "kubernetes-setup/master-playbook.yml":

  - name: copy pem file
    copy: src=BCPSG.pem dest=/etc/ssl/certs

  - name: Update cert index
    shell: /usr/sbin/update-ca-certificates 

where BCPSG.pm is the certificate I copied to the directory where the Vagrantfile is placed which is @ kubernetes-setup directory. referring back to https://kubernetes.io/blog/2019/03/15/kubernetes-setup-using-ansible-and-vagrant/

-- kamal
Source: StackOverflow

4/22/2019

Try remove $HOME/.kube directory and after kubeadm init give this commands again:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

-- Adi Soyadi
Source: StackOverflow

4/22/2019

remove the below param from init command

--node-name k8s-master

Include below param to deploy required kubernetes version

--kubernetes-version v1.14.1
-- P Ekambaram
Source: StackOverflow