My ultimate goal is to run Kubernetes on a 3-node CoreOS cluster (if anyone has better suggestions, I'm all ears: at this stage, I'm considering CoreOS a complete waste of my time).
I've started following the CoreOS guide to run a Vagrant cluster (I even have the CoreOs in Action book and that's not much help either) - I have obtained a fresh discovery token
and modified the user-data
and config.rb
files as described there:
#cloud-config
---
coreos:
etcd2:
discovery: https://discovery.etcd.io/7b9a3e994a14c2bf530ed88676e3fc97
and:
$ cat config.rb
# Size of the CoreOS cluster created by Vagrant
$num_instances = 3
$update_channel = "stable"
the rest is all as in the original coreos-vagrant
repository.
When first started, it appears that etcd
is not started as a service; starting it up with systemctl
seems to get it going, but it then does not discover its peers:
core@core-01 ~ $ etcdctl member list
8e9e05c52164694d: name=4adff068c464446a8423e9b9f7c28711 peerURLs=http://localhost:2380 clientURLs=http://localhost:2379 isLeader=true
the same on all other three Vagrant VMs.
It seems to me that either the modified user-data
is not picked up, or somehow the discovery token is ignored.
I have been googling around, but nothing seems to come up.
The main difficulty I'm finding is that virtually all the instructions around CoreOS/etcd point to these YAML files and then state one has to use ct
to generate the "real" configuration: but they don't really show how to do this on a running VM, or how to change the running configuration.
A couple of questions:
1) what would be the "right way" to get the three VMs' etcd
to start and finding each other? Reading up the guide here is really not that helpful.
The three VM are on a 'host-only' network:
core@core-01 ~ $ ip address show
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:76:a6:cf brd ff:ff:ff:ff:ff:ff
inet 172.17.8.101/16 brd 172.17.255.255 scope global eth1
(the other two are on 102
and 103
)
2) is there a "better" (for some definition of "better") way of getting a Kubernetes cluster running on 3 VirtualBox VMs?
It seems to me that CoreOS are trying to be too clever for their own good: I've been using various flavors of Linux for the past 10 years+ and getting the etcd cluster to find each other is proving frustratingly difficult.
I'm running Ubuntu 17.10 (well, all going well, it'll soon be 18.04 LTS) and Virtualbox 5.2.8.
Thanks in advance!
Unfortunately, the documentation you used is currently outdated. Right now ETCD version 3
is used as a Kubernetes
data storage. It provisions with Ignition
(VirtualBox Provider (default)):
When using the VirtualBox provider for Vagrant (the default), Ignition is used to provision the machine.
1. Install vagrant-ignition plugin (just in case if this plugin isn't automatically installed when using the default Vagrantfile from coreos-vagrant
repo):
git clone https://github.com/coreos/vagrant-ignition
cd vagrant-ignition
gem build vagrant-ignition.gemspec
vagrant plugin install vagrant-ignition-0.0.3.gem
2. Install ct.
3. Clone coreos-vagrant repo:
git clone https://github.com/coreos/coreos-vagrant
cd coreos-vagrant
4. Create config.rb
to start three CoreOS VMs:
cp config.rb.sample config.rb
sed -i 's/$num_instances=1/$num_instances=3/g' config.rb
5. Get etcd discovery token and put it into cl.conf
:
discovery_token=$(curl -s https://discovery.etcd.io/new\?size\=3)
sed -i "s|https://discovery.etcd.io/<token>|$discovery_token|g" cl.conf
6. Use config transpiler to write the Ignition config to config.ign
:
ct --platform=vagrant-virtualbox < cl.conf > config.ign
7. Create etcd cluster:
vagrant up
ETCD
cluster is ready:
core@core-01 ~ $ etcdctl member list
3655a3141d6f953b: name=core-01 peerURLs=http://172.17.8.101:2380 clientURLs=http://172.17.8.101:2379 isLeader=false
951a7a7a97c94116: name=core-02 peerURLs=http://172.17.8.102:2380 clientURLs=http://172.17.8.102:2379 isLeader=true
fd056871037fdb55: name=core-03 peerURLs=http://172.17.8.103:2380 clientURLs=http://172.17.8.103:2379 isLeader=false