How to setup Kubernetes Master HA on AWS

12/18/2015

What I am trying to do:

I have setup kubernete cluster using documentation available on Kubernetes website (http_kubernetes.io/v1.1/docs/getting-started-guides/aws.html). Using kube-up.sh, i was able to bring kubernete cluster up with 1 master and 3 minions (as highlighted in blue rectangle in the diagram below). From the documentation as far as i know we can add minions as and when required, So from my point of view k8s master instance is single point of failure when it comes to high availability.

Kubernetes Master HA on AWS

So I am trying to setup HA k8s master layer with the three master nodes as shown above in the diagram. For accomplishing this I am following kubernetes high availability cluster guide, http_kubernetes.io/v1.1/docs/admin/high-availability.html#establishing-a-redundant-reliable-data-storage-layer What I have done:

Setup k8s cluster using kube-up.sh and provider aws (master1 and minion1, minion2, and minion3) Setup two fresh master instance’s (master2 and master3) I then started configuring etcd cluster on master1, master 2 and master 3 by following below mentioned link: http_kubernetes.io/v1.1/docs/admin/high-availability.html#establishing-a-redundant-reliable-data-storage-layer So in short i have copied etcd.yaml from the kubernetes website (http_kubernetes.io/v1.1/docs/admin/high-availability/etcd.yaml) and updated Node_IP, Node_Name and Discovery Token on all the three nodes as shown below.

NODE_NAME NODE_IP DISCOVERY_TOKEN

Master1 172.20.3.150 https_discovery.etcd.io/5d84f4e97f6e47b07bf81be243805bed

Master2 172.20.3.200 https_discovery.etcd.io/5d84f4e97f6e47b07bf81be243805bed

Master3 172.20.3.250 https_discovery.etcd.io/5d84f4e97f6e47b07bf81be243805bed

And on running etcdctl member list on all the three nodes, I am getting:

$ docker exec <container-id> etcdctl member list
ce2a822cea30bfca: name=default peerURLs=http_localhost:2380,http_localhost:7001 clientURLs=http_127.0.0.1:4001

As per documentation we need to keep etcd.yaml in /etc/kubernete/manifest, this directory already contains etcd.manifest and etcd-event.manifest files. For testing I modified etcd.manifest file with etcd parameters.

After making above changes I forcefully terminated docker container, container was existing after few seconds and I was getting below mentioned error on running kubectl get nodes: error: couldn't read version from server: Get httplocalhost:8080/api: dial tcp 127.0.0.1:8080: connection refused

So please kindly suggest how can I setup k8s master highly available setup on AWS.

-- Neeraj Gupta
amazon-web-services
high-availability
kubernetes

3 Answers

6/21/2016

Setting up HA controllers for kubernetes is not trivial and I can't provide all the details here but I'll outline what was successful for me.

  1. Use kube-aws to set up a single-controller cluster: https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html. This will create CloudFormation stack templates and cloud-config templates that you can use as a starting point.
  2. Go the AWS CloudFormation Management Console, click the "Template" tab and copy out the complete stack configuration. Alternatively, use $ kube-aws up --export to generate the cloudformation stack file.
  3. User the userdata cloud-config templates generated by kube-aws and replace the variables with actual values. This guide will help you determine what those values should be: https://coreos.com/kubernetes/docs/latest/getting-started.html. In my case I ended up with four cloud-configs:
    • cloud-config-controller-0
    • cloud-config-controller-1
    • cloud-config-controller-2
    • cloud-config-worker
  4. Validate your new cloud-configs here: https://coreos.com/validate/
  5. Insert your cloud-configs into the CloudFormation stack config. First compress and encode your cloud config:

    $ gzip -k cloud-config-controller-0
    $ cat cloud-config-controller-0.gz | base64 > cloud-config-controller-0.enc

    Now copy the content into your encoded cloud-config into the CloudFormation config. Look for the UserData key for the appropriate InstanceController. (I added additional InstanceController objects for the additional controllers.)

  6. Update the stack at the AWS CloudFormation Management Console using your newly created CloudFormation config.

You will also need to generate TLS asssets: https://coreos.com/kubernetes/docs/latest/openssl.html. These assets will have to be compressed and encoded (same gzip and base64 as above), then inserted into your userdata cloud-configs.

When debugging on the server, journalctl is your friend:

$ journalctl -u oem-cloudinit  # to debug problems with your cloud-config
$ journalctl -u etcd2
$ journalctl -u kubelet

Hope that helps.

-- lander2k2
Source: StackOverflow

11/29/2016

There is also kops project

From the project README:

Operate HA Kubernetes the Kubernetes Way

also:

We like to think of it as kubectl for clusters

Download the latest release, e.g.:

cd ~/opt
wget https://github.com/kubernetes/kops/releases/download/v1.4.1/kops-linux-amd64
mv kops-linux-amd64 kops
chmod +x kops
ln -s ~/opt/kops ~/bin/kops

See kops usage, especially:

Assuming you already have s3://my-kops bucket and kops.example.com hosted zone.

Create configuration:

kops create cluster --state=s3://my-kops --cloud=aws \
    --name=kops.example.com \
    --dns-zone=kops.example.com \
    --ssh-public-key=~/.ssh/my_rsa.pub \
    --master-size=t2.medium \
    --master-zones=eu-west-1a,eu-west-1b,eu-west-1c \
    --network-cidr=10.0.0.0/22 \
    --node-count=3 \
    --node-size=t2.micro \
    --zones=eu-west-1a,eu-west-1b,eu-west-1c

Edit configuration:

kops edit cluster --state=s3://my-kops

Export terraform scripts:

kops update cluster --state=s3://my-kops --name=kops.example.com --target=terraform

Apply changes directly:

kops update cluster --state=s3://my-kops --name=kops.example.com --yes

List cluster:

kops get cluster --state s3://my-kops

Delete cluster:

kops delete cluster --state s3://my-kops --name=kops.identityservice.co.uk --yes
-- Paweł Prażak
Source: StackOverflow

1/28/2016

To configure an HA master, you should follow the High Availability Kubernetes Cluster document, in particular making sure you have replicated storage across failure domains and a load balancer in front of your replicated apiservers.

-- Robert Bailey
Source: StackOverflow