Kubernetes Node Cannot join cluster -TLS handshake error from 10.123.146.55:48344: remote error: bad certificate

11/29/2016

Trying to build 3-node Kubernetes cluster and running into and issue on CoreOS and I cannot join any nodes to cluster API Server keeps throwing an error regarding TLS certificate.

I1128 23:08:47.715663       1 logs.go:41] http: TLS handshake error from 10.123.146.55:48344: remote error: bad certificate
I1128 23:08:47.829082       1 logs.go:41] http: TLS handshake error from 10.123.146.55:48346: remote error: bad certificate
I1128 23:08:47.881655       1 logs.go:41] http: TLS handshake error from 10.123.146.54:40896: remote error: bad certificate
I1128 23:08:47.923955       1 logs.go:41] http: TLS handshake error from 10.123.146.54:40898: remote error: bad certificate

Certificate on the worker is valid - verified using

curl --key worker-key.pem -k https://10.123.146.53/api/v1/nodes --cert worker.pem  --cacert ca.pem

Only relevant error I could see is Controller is failing to start certificate controller:

I1128 22:45:12.452293       1 controllermanager.go:462] Starting certificates.k8s.io/v1alpha1 apis
I1128 22:45:12.452989       1 controllermanager.go:464] Starting certificate request controller
E1128 22:45:12.454607       1 controllermanager.go:474] Failed to start certificate controller: open /etc/kubernetes/ca/ca.pem: no such file or directory

From what i can tell /etc/kubernetes/is included from below:

$ cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
name: kube-controller-manager
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-controller-manager
  image: quay.io/coreos/hyperkube:v1.4.3_coreos.0
  command:
  - /hyperkube
  - controller-manager
  - --master=http://127.0.0.1:8080
  - --leader-elect=true
  - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
  - --root-ca-file=/etc/kubernetes/ssl/ca.pem
  livenessProbe:
    httpGet:
      host: 127.0.0.1
      path: /healthz
      port: 10252
    initialDelaySeconds: 15
    timeoutSeconds: 1
  volumeMounts:
  - mountPath: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
    readOnly: true
  - mountPath: /etc/ssl/certs
    name: ssl-certs-host
    readOnly: true
volumes:
- hostPath:
    path: /etc/kubernetes/ssl
  name: ssl-certs-kubernetes
- hostPath:
    path: /usr/share/ca-certificates
  name: ssl-certs-host
-- Uday Pandya
coreos
kubernetes

1 Answer

11/29/2016

The -k flag in curl ignores and hides TLS/SSL warnings. If you remove -k you will see this this is in fact a invalid TLS certificate.

From the curl manual:

-k, --insecure Allow connections to SSL sites without certs (H)

-- Hans Kristian
Source: StackOverflow