DataStoreError: Invalid ETCD_CA_CERT_FILE. Certificate Authority cert is required and must be a readable file path

12/12/2016

I have two CoreOS stable (1185.5.0) servers at home. I try to install kubernetes controller and worker on these two.

i use the coreos-kubernetes scripts to install from https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/generic, and patched from https://github.com/kfirufk/coreos-kubernetes-multi-node-generic-install-script. I use rkt to run the relevant containers.

I use the following environment variable options override file:

ETCD_AUTHORITY=coreos-3.tux-in.com:2379
ETCD_ENDPOINTS="https://coreos-2.tux-in.com:2379,https://coreos-3.tux-in.com:2379"
CONTROLLER_ENDPOINT=https://coreos-2.tux-in.com
K8S_VER=v1.5.0-beta.3_coreos.0
HYPERKUBE_IMAGE_REPO=quay.io/coreos/hyperkube
DNS_SERVICE_IP=10.3.0.10
USE_CALICO=true
CONTAINER_RUNTIME=rkt
OVERWRITE_ALL_FILES=true
ADVERTISE_IP=10.79.218.3
ETCD_CERT_FILE="/etc/ssl/etcd/etcd2.pem"
ETCD_KEY_FILE="/etc/ssl/etcd/etcd2-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/ssl/etcd/ca.pem"
ETCD_SCHEME="https"
IS_MASK_UPDATE_ENGINE=false

coreos-2.tux-in.com which resolves to 10.79.218.2 is the controller node.

coreos-3.tux-in.com which resolves to 10.79.218.3 is the worker node.

it seems that the controller script installs fine.

when I try to install the kubernetes worker on the 2nd server, I noticed the following error message keeps appearing in the kubelet log:

 2016-12-12 12:24:08,171 6960 [kube-system/kubernetes-dashboard-v1.4.1-kjj0c] ERROR Unhandled Exception killed plugin
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]: Traceback (most recent call last):
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   File "<string>", line 773, in main
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   File "<string>", line 64, in __init__
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   File "site-packages/pycalico/datastore.py", line 229, in __init__
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]: DataStoreError: Invalid ETCD_CA_CERT_FILE. Certificate Authority cert is required and must be a readable file path. Value provided:
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]: 2016-12-12 12:24:08,171 6960 [kube-system/kubernetes-dashboard-v1.4.1-kjj0c] ERROR CNI Error:
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]: {
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   "msg": "Unhandled Exception killed plugin",
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   "cniVersion": "0.1.0",
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   "code": 100,
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   "details": null
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]: }
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]: Traceback (most recent call last):
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   File "<string>", line 773, in main
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   File "<string>", line 64, in __init__
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]:   File "site-packages/pycalico/datastore.py", line 229, in __init__
Dec 12 12:24:08 coreos-3.tux-in.com kubelet-wrapper[1786]: DataStoreError: Invalid ETCD_CA_CERT_FILE. Certificate Authority cert is required and must be a readable file path. Value provided:

Invalid ETCD_CA_CERT_FILE error message shows that the value provided is empty, which shows that the ETCD_CA_CERT_FILE environment variable is not set for some reason. I tried editing /etc/systemd/system/kubelet.service and adding Environment=ETCD_CA_CERT_FILE=/etc/ssl/etcd/ca.pem under [Service] but the results are the same. any ideas ?

-- ufk
coreos
kubelet
kubernetes

1 Answer

12/14/2016

it appears there was a problem with parsing ETCD_CA_CERT_FILE parameter in calico node container. i found a bug report about it, can't find it now for some reason so can't paste here, sorry.

so anyhow using the latest calico-node version fixes the issue.

(Version v1.0.0-rc4 instead of 0.19.0)

-- ufk
Source: StackOverflow