I am trying to secure kubernetes, I have a master and a minion which both work, then I followed the guide at http://rootsquash.com/2016/05/10/securing-the-kubernetes-api/
I now have the running master which can be access through https but I was getting a "Unauthorized" error so I created a certificate for myself through the same process I did for the minion, created a p12 file which then imported into firefox, I restarted the browser and was prompted to authenticate with a certificate, I used the one I just imported and was presented with:
{
"major": "1",
"minor": "2",
"gitVersion": "v1.2.0",
"gitCommit": "ec7364b6e3b155e78086018aa644057edbe196e5",
"gitTreeState": "clean"
}
So I can now connect through the browser, then I go setup the minion and restart the service and when I check the status what I get is
kubelet[1655]: E1019 14:53:26.962906 1655 event.go:202] Unable to write event: 'x509: certificate signed by unknown authority' (may retry after sleeping)
I tried installing the root CA certificate I created in both the master and minion but that did not work so I thought maybe the certificate is corrupted, so using the same certificate the minion is using I did
curl -k --key /srv/kubernetes/${HOSTNAME}.key --cert /srv/kubernetes/${HOSTNAME}.crt --cacert /srv/kubernetes/ca.crt https://kubernetes.master:6443/version
And got the same
{
"major": "1",
"minor": "2",
"gitVersion": "v1.2.0",
"gitCommit": "ec7364b6e3b155e78086018aa644057edbe196e5",
"gitTreeState": "clean"
}
So the master is obviously rejecting my certificate for some other reason as using the certificate in curl works just fine, I been doing some googling but have not been able to fix this issue so far with what I have found, can anyone point me in the right direction?
My setup is on cent os minimal and the code used to generate the config file is below
kubectl config set-cluster my_cluster --server=https://kubernetes.master:6443 --insecure-skip-tls-verify=true
kubectl config unset clusters
kubectl config set-cluster my_cluster --certificate-authority=/srv/kubernetes/ca.crt --embed-certs=true --server=https://kubernetes.master:6443
kubectl config set-credentials minion --client-certificate=/srv/kubernetes/minion.one.crt --client-key=/srv/kubernetes/minion.one.key --embed-certs=true --token=59qbOTrcPcdSEgz1EZ0jOznJQ7uOYEsO
kubectl config set-context service-account-context --cluster=my_cluster --user=minion
kubectl config use-context service-account-context
[update]
Upon further checking it may well not even be related to the certificate or the tls handshake, I ran systemctl status -l kubelet.service And got the following
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.683943 13120 manager.go:123] Starting to sync pod status with apiserver
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.683958 13120 kubelet.go:2372] Starting kubelet main sync loop.
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.683967 13120 kubelet.go:2381] skipping pod synchronization - [container runtime is down]
Oct 19 16:11:58 minion.one kubelet[13120]: E1019 16:11:58.687984 13120 event.go:202] Unable to write event: 'Post https://kubernetes.master:6443/api/v1/namespaces/default/events: x509: certificate signed by unknown authority' (may retry after sleeping)
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.930635 13120 factory.go:233] Registering Docker factory
Oct 19 16:11:58 minion.one kubelet[13120]: I1019 16:11:58.930995 13120 factory.go:97] Registering Raw factory
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.063535 13120 manager.go:1003] Started watching for new ooms in manager
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.063556 13120 oomparser.go:198] OOM parser using kernel log file: "/var/log/messages"
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.063885 13120 manager.go:256] Starting recovery of all containers
Oct 19 16:11:59 minion.one kubelet[13120]: I1019 16:11:59.090785 13120 manager.go:261] Recovery completed
Could the first error
skipping pod synchronization - [container runtime is down]
Be causing the later issues with the certificate?
Trying to figure out where that error comes from
I ended up using this script which installs version 1.4
rm -f /etc/hostname
rm -f /etc/hosts
rm -f /etc/sysconfig/network
cat <<EOF > /etc/hostname
${HOSTNAME}
EOF
cat <<EOF > /etc/sysconfig/network
HOSTNAME=${HOSTNAME}
EOF
cat <<EOF > /etc/hosts
127.0.0.1 ${HOSTNAME} localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
${KUBERMASTER} kubernetes.vetology.net
EOF
systemctl disable firewalld
systemctl stop firewalld
yum -y update
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y docker kubelet kubeadm kubectl kubernetes-cni
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet
kubeadm init
All is working now.