Setting up release pipeline of kubernetes+istio with terraform+helm gives forbidden error

4/3/2019

I have tried now so many times setting up this pipeline in Azure devops where I want to deploy a AKS cluster and place istio on top.

Deploying the AKS using Terraform works great.

After this I try to install istio using helm but the command I use gives forbidden error.

helm.exe install --namespace istio-system --name istio-init --wait C:\Istio\install\kubernetes\helm\istio

I used the local path since this was the only good way I could find for helm to find the istio chart i have on the build agent.

The error message

Error: release istio-init failed: clusterroles.rbac.authorization.k8s.io "istio-galley-istio-system" is forbidden: attempt to grant extra privileges: [{[*] [admissionregistration.k8s.io] [validatingwebhookconfigurations] [] []} {[get] [config.istio.io] [*] [] []} {[list] [config.istio.io] [*] [] []} {[watch] [config.istio.io] [*] [] []} {[get] [*] [deployments] [istio-galley] []} {[get] [*] [endpoints] [istio-galley] []}] user=&{system:serviceaccount:kube-system:tillerserviceaccount 56632fa4-55e7-11e9-a4a1-9af49f3bf03a [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found, clusterroles.rbac.authorization.k8s.io "system:discovery" not found, clusterroles.rbac.authorization.k8s.io "cluster-admin" not found, clusterroles.rbac.authorization.k8s.io "system:discovery" not found, clusterroles.rbac.authorization.k8s.io "system:discovery" not found, clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]]

The serviceaccount I use (system:serviceaccount:kube-system:tillerserviceaccount as you can see in error message) are configured using this rbac config:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tillerserviceaccount
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tillerbinding
roleRef:
  apiGroup: ""
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tillerserviceaccount
    namespace: kube-system

Still the error message says in the ruleResolutionErrors that it looks for cluster-admin but it is not found.

I even tried the extreme and set all service accounts as cluster admins to test:

kubectl create clusterrolebinding serviceaccounts-admins --clusterrole=cluster-admin --group=system:serviceaccounts

But even after that I get the same error with the same ruleResolutionErrors.

I am stuck and appriciate any help in what I can do differently.

-- JohanSellberg
azure-aks
azure-kubernetes
istio
kubernetes
kubernetes-helm

1 Answer

4/3/2019

this is the role binding we are using in dev clusters:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tillerbinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tillerserviceaccount
    namespace: kube-system

edit: in this case the error was due to AKS created without RBAC.

-- 4c74356b41
Source: StackOverflow