CoreOS Cluster with Vagrant does not start/configure etcd correctly

4/2/2018

My ultimate goal is to run Kubernetes on a 3-node CoreOS cluster (if anyone has better suggestions, I'm all ears: at this stage, I'm considering CoreOS a complete waste of my time).

I've started following the CoreOS guide to run a Vagrant cluster (I even have the CoreOs in Action book and that's not much help either) - I have obtained a fresh discovery token and modified the user-data and config.rb files as described there:

#cloud-config

---
coreos:
  etcd2:
    discovery: https://discovery.etcd.io/7b9a3e994a14c2bf530ed88676e3fc97

and:

$ cat config.rb
# Size of the CoreOS cluster created by Vagrant
$num_instances = 3
$update_channel = "stable"

the rest is all as in the original coreos-vagrant repository.

When first started, it appears that etcd is not started as a service; starting it up with systemctl seems to get it going, but it then does not discover its peers:

core@core-01 ~ $ etcdctl member list
8e9e05c52164694d: name=4adff068c464446a8423e9b9f7c28711 peerURLs=http://localhost:2380 clientURLs=http://localhost:2379 isLeader=true

the same on all other three Vagrant VMs.

It seems to me that either the modified user-data is not picked up, or somehow the discovery token is ignored.

I have been googling around, but nothing seems to come up.

The main difficulty I'm finding is that virtually all the instructions around CoreOS/etcd point to these YAML files and then state one has to use ct to generate the "real" configuration: but they don't really show how to do this on a running VM, or how to change the running configuration.

A couple of questions:

1) what would be the "right way" to get the three VMs' etcd to start and finding each other? Reading up the guide here is really not that helpful.

The three VM are on a 'host-only' network:

core@core-01 ~ $ ip address show
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:76:a6:cf brd ff:ff:ff:ff:ff:ff
    inet 172.17.8.101/16 brd 172.17.255.255 scope global eth1

(the other two are on 102 and 103)

2) is there a "better" (for some definition of "better") way of getting a Kubernetes cluster running on 3 VirtualBox VMs?

It seems to me that CoreOS are trying to be too clever for their own good: I've been using various flavors of Linux for the past 10 years+ and getting the etcd cluster to find each other is proving frustratingly difficult.

I'm running Ubuntu 17.10 (well, all going well, it'll soon be 18.04 LTS) and Virtualbox 5.2.8.

Thanks in advance!

-- Marco Massenzio
coreos
etcd
kubernetes

1 Answer

4/4/2018

Unfortunately, the documentation you used is currently outdated. Right now ETCD version 3 is used as a Kubernetes data storage. It provisions with Ignition (VirtualBox Provider (default)):

When using the VirtualBox provider for Vagrant (the default), Ignition is used to provision the machine.

1. Install vagrant-ignition plugin (just in case if this plugin isn't automatically installed when using the default Vagrantfile from coreos-vagrant repo):

git clone https://github.com/coreos/vagrant-ignition
cd vagrant-ignition
gem build vagrant-ignition.gemspec
vagrant plugin install vagrant-ignition-0.0.3.gem

2. Install ct.

3. Clone coreos-vagrant repo:

git clone https://github.com/coreos/coreos-vagrant
cd coreos-vagrant

4. Create config.rb to start three CoreOS VMs:

cp config.rb.sample config.rb
sed -i 's/$num_instances=1/$num_instances=3/g' config.rb

5. Get etcd discovery token and put it into cl.conf:

discovery_token=$(curl -s https://discovery.etcd.io/new\?size\=3)
sed -i "s|https://discovery.etcd.io/<token>|$discovery_token|g" cl.conf

6. Use config transpiler to write the Ignition config to config.ign:

ct --platform=vagrant-virtualbox < cl.conf > config.ign

7. Create etcd cluster:

vagrant up

ETCD cluster is ready:

core@core-01 ~ $ etcdctl member list
3655a3141d6f953b: name=core-01 peerURLs=http://172.17.8.101:2380 clientURLs=http://172.17.8.101:2379 isLeader=false
951a7a7a97c94116: name=core-02 peerURLs=http://172.17.8.102:2380 clientURLs=http://172.17.8.102:2379 isLeader=true
fd056871037fdb55: name=core-03 peerURLs=http://172.17.8.103:2380 clientURLs=http://172.17.8.103:2379 isLeader=false
-- nickgryg
Source: StackOverflow