K
Q

No tiller when installing Helm with terraform in a K8S cluster on DigitalOcean

10/4/2019

I'm trying to deploy a new cluster on DigitalOcean using Terraform, the main idea is to deploy Fission to have a serverless environment, here are my files:

do_provider.tf

provider "digitalocean" {
  token = "${var.do_token}"
}

variables.tf

variable region {
  description = "Region for cloud resources."
  default     = "lon1"
}

k8s_cluster.tf

resource "random_id" "fission-cluster" {
  byte_length = 8
}

resource "digitalocean_kubernetes_cluster" "fission-cluster" {
  name = "k8s-fission-${var.region}-${random_id.fission-cluster.hex}"
  region = "${var.region}"
  version =  "1.15.3-do.3"
  node_pool {
    name = "k8s-fission-${var.region}-${random_id.fission-cluster.hex}-worker"
    size = "s-1vcpu-2gb"
    node_count = "3"
    tags = ["fission", "worker"]
  }
  tags = ["fission"]
}

resource "local_file" "config" {
    content     = "${digitalocean_kubernetes_cluster.fission-cluster.kube_config.0.raw_config}"
    filename = "${path.module}/config"
}

provider "kubernetes" {
  host = "${digitalocean_kubernetes_cluster.fission-cluster.endpoint}"

  client_certificate     = "${base64decode(digitalocean_kubernetes_cluster.fission-cluster.kube_config.0.client_certificate)}"
  client_key             = "${base64decode(digitalocean_kubernetes_cluster.fission-cluster.kube_config.0.client_key)}"
  cluster_ca_certificate = "${base64decode(digitalocean_kubernetes_cluster.fission-cluster.kube_config.0.cluster_ca_certificate)}"
}

helm.tf

provider "helm" {
    service_account = "tiller"
    install_tiller = true
    namespace = "kube-system"
    kubernetes {
      config_path =  "${path.module}/config"
      load_config_file = true
    }
}

data "helm_repository" "stable" {
  name = "stable"
  url  = "https://kubernetes-charts.storage.googleapis.com"
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }
  depends_on =  [ "kubernetes_cluster_role_binding.tiller" ]
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller"
  }
  role_ref {
      api_group = "rbac.authorization.k8s.io"
      kind = "ClusterRole"
      name = "cluster-admin"
  }
  subject {
      kind = "User"
      name = "admin"
      api_group = "rbac.authorization.k8s.io"
  }
  subject {
      kind = "ServiceAccount"
      name = "tiller"
      namespace = "kube-system"
  }
  subject {
      kind = "Group"
      name = "system:masters"
      api_group = "rbac.authorization.k8s.io"
  }
}

When I run terrarform plan it show the changes correctlly and also when I run terraform apply cluster is created and shows the output as correct

random_id.fission-cluster: Creating...
random_id.fission-cluster: Creation complete after 0s [id=dag8ooN_yVg]
digitalocean_kubernetes_cluster.fission-cluster: Creating...
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [10s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [20s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [30s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [40s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [50s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [1m0s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [1m10s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [1m20s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [1m30s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [1m40s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [1m50s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [2m0s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [2m10s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [2m20s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [2m30s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [2m40s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [2m50s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [3m0s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [3m10s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [3m20s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [3m30s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [3m40s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [3m50s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [4m0s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [4m10s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [4m20s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [4m30s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [4m40s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [4m50s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [5m0s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [5m10s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [5m20s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [5m30s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [5m40s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Still creating... [5m50s elapsed]
digitalocean_kubernetes_cluster.fission-cluster: Creation complete after 5m52s [id=a2dbf847-a273-41c0-a5f7-5aab8ab21407]
local_file.config: Creating...
local_file.config: Creation complete after 0s [id=bab7483e4abd5e02e473464556055c80ec952826]
kubernetes_service_account.tiller: Creating...
kubernetes_service_account.tiller: Creation complete after 2s [id=kube-system/tiller]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

The problem is that after all the process the tiller is not deployed inside the cluster and the helm cli command request to initialize the tiller first.

UPDATE

The helm.tf file was wrong

-- wolmi
digital-ocean
kubernetes
kubernetes-helm
terraform

Similar Questions

Monitoring EKS Kubernetes LoadBalancer service Type
How to create redirection url
Kubectl missing form Terraform Cloud
Run kubectl cronjob reusing a deployment template
What API endpoint to call to get a Kubernetes pod or service's yaml?

1 Answer

10/4/2019

The tiller component should be installed by default with the helm terraform provider. However it looks for the kube config file by default at path:

~/.kube/config

In your configuration, it looks like you're saving the new Kubernetes cluster kube config into the module path with your local_file config resource.

i.e.

"${path.module}/config"

So I would adjust your kubernetes block in the helm provider section to add something like:

load_config_file = "${path.module}/config"

Failing that, enable a higher log debug level when running - perhaps the failure to initialize the helm tiller component is a silent warning or info log message.

Try set the TF_LOG environment variable to TRACE. Debugging docs for Terraform

-- Shogan
Source: StackOverflow