Ambassador pod crashed with error "message":"namespaces \"default\" is forbidden

6/19/2020

Ambassador link which was previously working stopped working. Ambassador is installed in namespace platform-ns.

kubectl logs ambassador-6fb4f95889-pjr6b -n platform-ns

2020-06-19 09:14:25 kubewatch [10 TMainThread] 0.75.0 DEBUG: looking up ID for namespace default
2020-06-19 09:14:25 kubewatch [10 TMainThread] 0.75.0 ERROR: couldn't read namespace default? (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6a4514c1-e475-4445-9994-ccb29f0e9bc1', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Fri, 19 Jun 2020 09:14:25 GMT', 'Content-Length': '335'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"default\" is forbidden: User \"system:serviceaccount:platform-ns:ambassador\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"default","kind":"namespaces"},"code":403}
2020/06/19 09:14:30 ERROR(s): kubebootstrap: WORKER PANICKED: mappings.getambassador.io is forbidden: User "system:serviceaccount:platform-ns:ambassador" cannot list resource "mappings" in API group "getambassador.io" at the cluster scope
goroutine 102 [running]:
![Text]Already tried running:
kubectl create serviceaccount --namespace platform-ns tiller
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=platform-ns:tiller
kubectl patch deploy --namespace platform-ns tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
-- Sumana
ambassador
azure-aks
kubernetes
kubernetes-helm
kubernetes-pod

1 Answer

6/24/2020

This looks to be an RBAC issue. Check through the ambassador ClusterRole, ClusterRoleBinding, and ServiceAccount to make sure they all sync up. The ServiceAccount should be in the same ns as Ambassador and should have a default token assigned to it. If you only have single namespace permissions, you can actually do that in helm with the scope.singleNamespace: true helm value.

-- Casey
Source: StackOverflow