can't setup kubernetes using a certificate as x509: certificate signed by unknown authority

10/19/2016

I am trying to secure kubernetes, I have a master and a minion which both work, then I followed the guide at http://rootsquash.com/2016/05/10/securing-the-kubernetes-api/

I now have the running master which can be access through https but I was getting a "Unauthorized" error so I created a certificate for myself through the same process I did for the minion, created a p12 file which then imported into firefox, I restarted the browser and was prompted to authenticate with a certificate, I used the one I just imported and was presented with:

{
  "major": "1",
  "minor": "2",
  "gitVersion": "v1.2.0",
  "gitCommit": "ec7364b6e3b155e78086018aa644057edbe196e5",
  "gitTreeState": "clean"
}

So I can now connect through the browser, then I go setup the minion and restart the service and when I check the status what I get is

kubelet[1655]: E1019 14:53:26.962906    1655 event.go:202] Unable to write event: 'x509: certificate signed by unknown authority' (may retry after sleeping)

I tried installing the root CA certificate I created in both the master and minion but that did not work so I thought maybe the certificate is corrupted, so using the same certificate the minion is using I did

 curl -k --key /srv/kubernetes/${HOSTNAME}.key  --cert /srv/kubernetes/${HOSTNAME}.crt  --cacert /srv/kubernetes/ca.crt https://kubernetes.master:6443/version

And got the same

{
  "major": "1",
  "minor": "2",
  "gitVersion": "v1.2.0",
  "gitCommit": "ec7364b6e3b155e78086018aa644057edbe196e5",
  "gitTreeState": "clean"
}

So the master is obviously rejecting my certificate for some other reason as using the certificate in curl works just fine, I been doing some googling but have not been able to fix this issue so far with what I have found, can anyone point me in the right direction?

My setup is on cent os minimal and the code used to generate the config file is below

kubectl config set-cluster my_cluster --server=https://kubernetes.master:6443 --insecure-skip-tls-verify=true
kubectl config unset clusters
kubectl config set-cluster my_cluster --certificate-authority=/srv/kubernetes/ca.crt --embed-certs=true --server=https://kubernetes.master:6443
kubectl config set-credentials minion --client-certificate=/srv/kubernetes/minion.one.crt --client-key=/srv/kubernetes/minion.one.key --embed-certs=true --token=59qbOTrcPcdSEgz1EZ0jOznJQ7uOYEsO
kubectl config set-context service-account-context --cluster=my_cluster --user=minion
kubectl config use-context service-account-context

[update]

Upon further checking it may well not even be related to the certificate or the tls handshake, I ran systemctl status -l kubelet.service And got the following

Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.683943   13120 manager.go:123] Starting to sync pod status with apiserver
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.683958   13120 kubelet.go:2372] Starting kubelet main sync loop.
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.683967   13120 kubelet.go:2381] skipping pod synchronization - [container runtime is down]
Oct 19 16:11:58 minion.one kubelet[13120]: E1019 16:11:58.687984   13120 event.go:202] Unable to write event: 'Post https://kubernetes.master:6443/api/v1/namespaces/default/events: x509: certificate signed by unknown authority' (may retry after sleeping)
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.930635   13120 factory.go:233] Registering Docker factory
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.930995   13120 factory.go:97] Registering Raw factory
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.063535   13120 manager.go:1003] Started watching for new ooms in manager
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.063556   13120 oomparser.go:198] OOM parser using kernel log file: "/var/log/messages"
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.063885   13120 manager.go:256] Starting recovery of all containers
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.090785   13120 manager.go:261] Recovery completed

Could the first error

skipping pod synchronization - [container runtime is down]

Be causing the later issues with the certificate?

Trying to figure out where that error comes from

-- Tlacaelel Ramon Luis
kubernetes
networking
x509

1 Answer

10/20/2016

I ended up using this script which installs version 1.4

rm -f /etc/hostname
rm -f /etc/hosts
rm -f /etc/sysconfig/network
cat <<EOF > /etc/hostname
${HOSTNAME}
EOF
cat <<EOF > /etc/sysconfig/network
HOSTNAME=${HOSTNAME}
EOF
cat <<EOF > /etc/hosts
127.0.0.1   ${HOSTNAME} localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
${KUBERMASTER} kubernetes.vetology.net
EOF
systemctl disable firewalld
systemctl stop firewalld
yum -y update
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y docker kubelet kubeadm kubectl kubernetes-cni
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet
kubeadm init

All is working now.

-- Tlacaelel Ramon Luis
Source: StackOverflow