K
Q

Terraform apply, how to increment count and add kubernetes worker nodes to the existing workers?

11/13/2019

I deployed k8s cluster on bare metal using terraform following this repository on github

Now I have three nodes:

ewr1-controller, ewr1-worker-0, ewr1-worker-1

Next, I would like to run terraform apply and increment the worker nodes (*ewr1-worker-3, ewr1-worker-4 ... *) while keeping the existing controller and worker nodes. I tried incrementing the count.index to start from 3, however it still overwrites the existing workers.

resource "packet_device" "k8s_workers" {
  project_id       = "${packet_project.kubenet.id}"
  facilities       = "${var.facilities}"
  count            = "${var.worker_count}"
  plan             = "${var.worker_plan}"
  operating_system = "ubuntu_16_04"
  hostname         = "${format("%s-%s-%d", "${var.facilities[0]}", "worker", count.index+3)}"
  billing_cycle    = "hourly"
  tags             = ["kubernetes", "k8s", "worker"]
}

I havent tried this but if I do

terraform state rm 'packet_device.k8s_workers'

I am assuming these worker nodes will not be managed by the kubernetes master. I don't want to create all the nodes at beginning because the worker nodes that I am adding will have different specs(instance types).

The entire script I used is available here on this github repository. I appreciate it if someone could tell what I am missing here and how to achieve this.

Thanks!

-- Zstack
bare-metal-server
kubeadm
kubernetes
metallb
terraform

Similar Questions

Nginx Ingress - how to set redirect to external site if there is no marching route?
nginx ingress 504 timeout with running multi pod
certificate is valid for ingress.local, not gitlab.mydomain
Nginx routing ignores anything after path rule

2 Answers

11/18/2019

Node resizing is best addressed using an autoscaler. Using Terraform to scale a nodepool might not be the optimal approach as the tool is meant to declare the state of the system rather than dynamically change it. The best approach for this is to use a cloud auto scaler. In bare metal, you can implement a CloudProvider interface (like the one provided by cloud such as AWS, GCP, Azure) as described here

After implementing that, you need to determine if your K8s implementation can be operated as a provider by Terraform, and if that's the case, find the nodepool autoscaler resource that allows the autoscaling.

Wrapping up, Terraform is not meant to be used as an autoscaler given its natures as a declarative language that describes the infrastructure.

The autoscaling features in K8s are meant to tackle this kind of requirements.

-- acid_fuji
Source: StackOverflow
11/18/2019

I solved this issue by modifying and removing modules, resources from the terraform.state: manually modifying it and using terraform state rm <--->.

  • Leaving out (remove state) the section that I want to keep as they are.

  • Modifying the sections in terraform.state that I want change when new servers are added.

  • Incrementing the counter to add new resources, see terraform interpolation.

I am using bare-metal cloud provider to deploy k8s and it doesn't support k8s HA or VA autos-calling. This is may not be optimal solution as other have pointed out but if it is not something you need to do quite often, terraform can do the job albeit the hardway.

-- Zstack
Source: StackOverflow