Accessing Kubernetes Masternode via kubectl from inside a pod results in HTTP 403

6/4/2019

I have tried to install an azure build agent on our on-prem kubernetes cluster. When trying to run builds, we always hit Status 403 Forbidden when executing kubectl commands.

Since RBAC is enabled, I have setup a ServiceAccount, CluterRole and ClusterRoleBinding for our build agent namespace with the following rules:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: build-agent
  namespace: agent_ns
automountServiceAccountToken: false
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: build-agent
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
      - nodes
      - nodes/proxy
      - proxy
      - pods
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: build-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: build-agent
subjects:
- kind: ServiceAccount
  name: build-agent
  namespace: agent_ns

Still, if I sh into the pod using:

kubectl exec -it build-agent-xxxxxxxx-xxxx --namespace=agent_ns -- sh

And try to access the kubernetes api via curl I get 403:

curl https://10.x.x.x:6443
curl: (56) Received HTTP code 403 from proxy after CONNECT

The kubeconfig file is present in the pod VM. It is the same I use from the host machine when I kubectl to the cluster. Same as the curl command, if I try to run a kubectl command after sh-ing into the pod, the requests returns 403:

> kubectl get ns
Unable to connect to the server: Forbidden

EDIT:

Deployment description now also contains a service accout specification. Problem still persists:

> kubectl get pod build-agent-xxxxxxxxx-xxxxx -o yaml -n agent_ns

 ...
 spec:
    automountServiceAccountToken: false
 ...
 serviceAccount: build-agent
 serviceAccountName: build-agent

Am I missing something? I can provide more information if necessary, just let me know.

The cluster is running fine, we already have tiller, traefik and a service which is up and running.

-- DeiAndrei
azure-devops
kubectl
kubernetes
on-prem

0 Answers