authentication handshake failed: x509: certificate signed by unknown authority

8/24/2019

I am starting kubernetes api server(v1.15.3) using this command:

systemctl start kube-apiserver.service

this is the log output:

kube-apiserver.service - Kubernetes API Service
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: activating (start) since 六 2019-08-24 20:12:18 CST; 4s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 9563 (kube-apiserver)
    Tasks: 13
   Memory: 11.0M
   CGroup: /system.slice/kube-apiserver.service
           └─9563 /usr/local/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=https://172.19.104.231:2379,https://172.19.104.230:2379,https://172.19.150.82:2379 --advertise-address=172.19.104.231 --bind-address=172.19.104.231 --insecure-bind-address=172.19.104.231 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h

824 20:12:19 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:19.994504    9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.150.82:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...
824 20:12:20 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:20.985988    9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.104.231:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...
824 20:12:20 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:20.986331    9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.104.230:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...

this CA certificate of kubernetes config(kubernetes-csr.json):

{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "172.19.104.230",
      "172.19.150.82",
      "172.19.104.231"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

What should I do to fix this problem?I have tried self sign certificate in CentOS 7:

openssl x509 -outform der -in kubernetes.pem -out kubernetes.crt
cp /data/k8s/ssl/kubernetes.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust

My etcd cluster using the same certification file.This is the generate certificate command:

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

this is the etcd list:

[root@iZuf63refzweg1d9dh94t8Z ssl]# etcdctl member list
55a782166ce91d01, started, infra3, https://172.19.150.82:2380, https://172.19.150.82:2379
67bca27e43a8258a, started, infra2, https://172.19.104.230:2380,
696a771758a889c4, started, infra1, https://172.19.104.231:2380, https://172.19.104.231:2379
-- Dolphin
kubernetes

1 Answer

8/25/2019

This may caused by your certificate file generate encount warning,you should use new version of cfssl(above v1.2),and make sure have no warning.This is cause by this tip when using cfssl(v1.3) to generate certificate:

This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements")

try to upgrade the cfssl to v1.3.4 and regenerate certificate.

/usr/local/go/bin/go get -u github.com/cloudflare/cfssl/cmd/cfssl

verify the version.

[root@iZuf63refzweg1d9dh94t8Z ssl]# /root/go/bin/cfssl version
Version: 1.3.4
Revision: dev
Runtime: go1.12.9
-- Dolphin
Source: StackOverflow