ETCD 3.4.3 not starting with systemd file

11/4/2019

I am trying to run a single node ETCD 3.4.3 cluster. I used to run ETCD cluster with the same configuration file before but now the configuration is not working for any version of ETCD. The ETCD server is a virtualbox(6.0.14r133895) machine spawned using Vargrant(2.2.6)

I use the following systemd file for ETCD

[Unit]
Description=Etcd Server
Documentation=https://github.com/coreos/etcd
After=network.target

[Service]
User=etcd
Type=notify
EnvironmentFile=/etc/default/etcd.conf
ExecStart=/usr/local/bin/etcd

Restart=always
RestartSec=10s
LimitNOFILE=40000
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

The environment file for ETCD configuration is as below

$ cat /etc/default/etcd.conf
ETCD_NAME=k8s-server-1
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://k8s-server-1:2380
ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
ETCD_ADVERTISE_CLIENT_URLS=https://k8s-server-1:2379,https://127.0.0.1:2379
ETCD_DISCOVERY=https://discovery.etcd.io/78dcc2d35f9a2c1d60f3d457d5a36bfe
ETCD_DISCOVERY_FALLBACK='exit'
ETCD_DATA_DIR=/var/lib/etcd/data
ETCD_WAL_DIR=/var/lib/etcd/wal
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/tls/k8s/ca/ca.pem
ETCD_CERT_FILE=/etc/tls/k8s/etcd/k8s-server-1-client.pem
ETCD_KEY_FILE=/etc/tls/k8s/etcd/k8s-server-1-client-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/tls/k8s/ca/ca.pem
ETCD_PEER_CERT_FILE=/etc/tls/k8s/etcd/k8s-server-1-peer.pem
ETCD_PEER_KEY_FILE=/etc/tls/k8s/etcd/k8s-server-1-peer-key.pem
ETCD_HEARTBEAT_INTERVAL=6000
ETCD_ELECTION_TIMEOUT=30000
ETCD_ENABLE_V2=false
ETCD_LOG_LEVEL=debug
GOMAXPROCS=2

Using the above config, ETCD fails to start with the following error

Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://k8s-server-1:2379,https://127.0.0.1:2379
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_CERT_FILE=/etc/tls/k8s/etcd/k8s-server-1-client.pem
Nov 04 05:41:28 k8s-server-1 etcd[15760]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Nov 04 05:41:28 k8s-server-1 etcd[15760]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/data
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/78dcc2d35f9a2c1d60f3d457d5a36bfe
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_DISCOVERY_FALLBACK=exit
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=30000
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_ENABLE_V2=false
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=6000
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://k8s-server-1:2380
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_KEY_FILE=/etc/tls/k8s/etcd/k8s-server-1-client-key.pem
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_LOG_LEVEL=debug
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_NAME=k8s-server-1
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/tls/k8s/etcd/k8s-server-1-peer.pem
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/tls/k8s/etcd/k8s-server-1-peer-key.pem
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/tls/k8s/ca/ca.pem
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/tls/k8s/ca/ca.pem
Nov 04 05:41:28 k8s-server-1 etcd[15760]: recognized and used environment variable ETCD_WAL_DIR=/var/lib/etcd/wal
Nov 04 05:41:28 k8s-server-1 etcd[15760]: etcd Version: 3.4.3
Nov 04 05:41:28 k8s-server-1 etcd[15760]: Git SHA: 3cf2f69b5
Nov 04 05:41:28 k8s-server-1 etcd[15760]: Go Version: go1.12.12
Nov 04 05:41:28 k8s-server-1 etcd[15760]: Go OS/Arch: linux/amd64
Nov 04 05:41:28 k8s-server-1 etcd[15760]: setting maximum number of CPUs to 2, total number of available CPUs is 2
Nov 04 05:41:28 k8s-server-1 etcd[15760]: the server is already initialized as member before, starting as etcd member...
Nov 04 05:41:28 k8s-server-1 etcd[15760]: peerTLS: cert = /etc/tls/k8s/etcd/k8s-server-1-peer.pem, key = /etc/tls/k8s/etcd/k8s-server-1-peer-key.pem, trusted-ca = /etc/tls/k8s/ca/ca.pem, client-cert-auth = true, crl-file = 
Nov 04 05:41:28 k8s-server-1 etcd[15760]: name = k8s-server-1
Nov 04 05:41:28 k8s-server-1 etcd[15760]: data dir = /var/lib/etcd/data
Nov 04 05:41:28 k8s-server-1 etcd[15760]: member dir = /var/lib/etcd/data/member
Nov 04 05:41:28 k8s-server-1 etcd[15760]: dedicated WAL dir = /var/lib/etcd/wal
Nov 04 05:41:28 k8s-server-1 etcd[15760]: heartbeat = 6000ms
Nov 04 05:41:28 k8s-server-1 etcd[15760]: election = 30000ms
Nov 04 05:41:28 k8s-server-1 etcd[15760]: snapshot count = 100000
Nov 04 05:41:28 k8s-server-1 etcd[15760]: discovery URL= https://discovery.etcd.io/78dcc2d35f9a2c1d60f3d457d5a36bfe
Nov 04 05:41:28 k8s-server-1 etcd[15760]: advertise client URLs = https://127.0.0.1:2379,https://k8s-server-1:2379
Nov 04 05:41:28 k8s-server-1 etcd[15760]: {"level":"info","ts":1572846088.8206353,"caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://k8s-server-1:2380","host":"k8s-server-1:2380","resolved-addr":"127.0.1.1:2380"}
Nov 04 05:41:28 k8s-server-1 etcd[15760]: {"level":"info","ts":1572846088.8212357,"caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://k8s-server-1:2380","host":"k8s-server-1:2380","resolved-addr":"127.0.1.1:2380"}
Nov 04 05:41:33 k8s-server-1 etcd[15760]: failed to join discovery cluster (context deadline exceeded)
Nov 04 05:41:33 k8s-server-1 etcd[15760]: discovery token https://discovery.etcd.io/78dcc2d35f9a2c1d60f3d457d5a36bfe was used, but failed to bootstrap the cluster.
Nov 04 05:41:33 k8s-server-1 etcd[15760]: please generate a new discovery token and try to bootstrap again.
Nov 04 05:41:33 k8s-server-1 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 05:41:33 k8s-server-1 systemd[1]: etcd.service: Failed with result 'exit-code'.
Nov 04 05:41:33 k8s-server-1 systemd[1]: Failed to start Etcd Server.
-- Subject: Unit etcd.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit etcd.service has failed.

However, I am able to start ETCD manually as below

$ sudo su etcd
etcd@k8s-server-1:~$ etcd --name $ETCD_NAME   \
    --initial-advertise-peer-urls $ETCD_INITIAL_ADVERTISE_PEER_URLS  \
    --advertise-client-urls $ETCD_ADVERTISE_CLIENT_URLS  \
    --client-cert-auth \
    --trusted-ca-file $ETCD_TRUSTED_CA_FILE  \
    --cert-file $ETCD_CERT_FILE  \
    --key-file $ETCD_KEY_FILE  \
    --peer-client-cert-auth  \
    --peer-trusted-ca-file $ETCD_PEER_TRUSTED_CA_FILE  \
    --peer-cert-file $ETCD_PEER_CERT_FILE  \
    --peer-key-file $ETCD_PEER_KEY_FILE  \
    --listen-peer-urls $ETCD_LISTEN_PEER_URLS  \
    --listen-client-urls $ETCD_LISTEN_CLIENT_URLS \
    --initial-cluster-token $ETCD_INITIAL_CLUSTER_TOKEN 

Once I start ETCD manually, the systemd unit file continue to work with the existing configuration. Can anyone please tell me why ETCD fails to start with the systemd file at the beginning.

-- zaman sakib
etcd
kubernetes
systemd

1 Answer

11/5/2019

Solved the problem removing discovery service to create the cluster. Instead created the cluster with specific node's IP. Also it seems like discovery mechanism behaves abnormally if internet speed is not fast enough.

-- zaman sakib
Source: StackOverflow