K
Q

Accessing Kubernetes API from pod fails although roles are configured is configured

8/30/2018

I am trying to issue letsencrypt cerficicates for a kubernetes cluster deployments using a solution inspired from here.

Ultimately I execute the following curl statement from a pod to update a cluster resource:

curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -k -v -XPATCH -H "Accept: application/json, */*" -H "Content-Type: application/strategic-merge-patch+json" -d @secret-patch.yml https://kubernetes/api/v1/namespaces/${NAMESPACE}/secrets/${SECRET}

To get the following error:

"message": "secrets \"converse-hr\" is forbidden: User \"system:serviceaccount:default:letsencrypt\" cannot patch secrets in the namespace \"default\": Unknown user \"system:serviceaccount:default:letsencrypt\"",

I have followed the instructions in these tickets [1,2,3].

And I have the following configuration and output:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: letsencrypt-service-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: letsencrypt-service-dev
  template:
    metadata:
      labels:
        app: letsencrypt-service-dev
    spec:
      serviceAccountName: letsencrypt
      containers:
      - name: letsencrypt-service-dev
        image: <image-name>
        ports:
        - name: letsencrypt
          containerPort: 3300
        imagePullPolicy: Always
        env:
          - name: DOMAINS
            value: <domain>
          - name: EMAIL
            value: <email>
          - name: SECRET
            value: <secret-name>
          - name: DEPLOYMENT
            value: <deployment>
---

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: letsencrypt

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: patch-tls
  namespace: default
subjects:
- kind: Group
  name: system:serviceaccount:default:letsencrypt # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole #this must be Role or ClusterRole
  name: cluster-admin # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

Here is the output of relevant kubectl commands

kubectl describe serviceaccount letsencrypt
Name:         letsencrypt
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"letsencrypt","namespace":"default"}}

Image pull secrets:  gcr-base-registry-json-key
Mountable secrets:   letsencrypt-token-xdp7w
Tokens:              letsencrypt-token-xdp7w
Events:              <none>

kubectl describe clusterrolebinding patch-tls
Name:         patch-tls
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"patch-tls","namespace":""},"roleRef":{"ap...
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind   Name                                       Namespace
  ----   ----                                       ---------
  Group  system:serviceaccount:default:letsencrypt
kubectl describe pod letsencrypt-service-dev-7f989dddc4-bxkss
Name:           letsencrypt-service-dev-7f989dddc4-bxkss
Namespace:      default
Node:           <node-name>
Start Time:     Thu, 30 Aug 2018 10:03:58 +0200
Labels:         app=letsencrypt-service-dev
                pod-template-hash=3954588870
Annotations:    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container letsencrypt-service-dev
Status:         Running
IP:             10.32.2.21
Controlled By:  ReplicaSet/letsencrypt-service-dev-7f989dddc4
Containers:
  letsencrypt-service-dev:
    Container ID:   docker://e6cd039cae94b564d3a2d5cf2768f363abe558118010c439772823f16c9a9a32
    Image:          <image-name>
    Image ID:       <image-id>
    Port:           3300/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 30 Aug 2018 10:04:00 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      DOMAINS:     <domain>
      EMAIL:       <email>
      SECRET:      <secret>
      DEPLOYMENT:  <deployment>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from letsencrypt-token-xdp7w (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  letsencrypt-token-xdp7w:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  letsencrypt-token-xdp7w
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
kubectl version                                              
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-08T16:31:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.5", GitCommit:"9b635efce81582e1da13b35a7aa539c0ccb32987", GitTreeState:"clean", BuildDate:"2018-08-02T23:42:40Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Working set of resources based on the answers below:

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: letsencrypt
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: patch-tls
  namespace: default
subjects:
- kind: ServiceAccount
  name: letsencrypt # Name is case sensitive
  namespace: default
roleRef:
  kind: ClusterRole #this must be Role or ClusterRole
  name: cluster-admin # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io
kubectl describe pod letsencrypt-service-dev-7f989dddc4-bxkss
    Name:           letsencrypt-service-dev-7f989dddc4-bxkss
    Namespace:      default
    Node:           <node-name>
    Start Time:     Thu, 30 Aug 2018 10:03:58 +0200
    Labels:         app=letsencrypt-service-dev
                    pod-template-hash=3954588870
    Annotations:    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container letsencrypt-service-dev
    Status:         Running
    IP:             10.32.2.21
    Controlled By:  ReplicaSet/letsencrypt-service-dev-7f989dddc4
    Containers:
      letsencrypt-service-dev:
        Container ID:   docker://e6cd039cae94b564d3a2d5cf2768f363abe558118010c439772823f16c9a9a32
        Image:          <image-name>
        Image ID:       <image-id>
        Port:           3300/TCP
        Host Port:      0/TCP
        State:          Running
          Started:      Thu, 30 Aug 2018 10:04:00 +0200
        Ready:          True
        Restart Count:  0
        Requests:
          cpu:  100m
        Environment:
          DOMAINS:     <domain>
          EMAIL:       <email>
          SECRET:      <secret>
          DEPLOYMENT:  <deployment>
        Mounts:
          /var/run/secrets/kubernetes.io/serviceaccount from letsencrypt-token-xdp7w (ro)
    Conditions:
      Type           Status
      Initialized    True 
      Ready          True 
      PodScheduled   True 
    Volumes:
      letsencrypt-token-xdp7w:
        Type:        Secret (a volume populated by a Secret)
        SecretName:  letsencrypt-token-xdp7w
        Optional:    false
    QoS Class:       Burstable
    Node-Selectors:  <none>
    Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s
    Events:          <none>
-- Vasil K
google-kubernetes-engine
kubernetes

Similar Questions

How do I configure my DNS to work with Rancher 2.0 ingress?
Openshift Origin 1.5.1 Pod anti-affinity on DeploymentConfig not working
Stackdriver - How to fetch logs of a kubernetes pod running on Google Cloud using REST APIs?
Nginx remove a path in uri
NodePort type service not accessible outside cluster

2 Answers

8/30/2018

I suppose serviceaccount can't be configured as group, try the following:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: patch-tls
subjects:
- kind: ServiceAccount
  name: letsencrypt
  namespace: default
roleRef:
  kind: ClusterRole 
  name: cluster-admin 
  apiGroup: rbac.authorization.k8s.io
-- Kun Li
Source: StackOverflow
8/30/2018

Try binding the role and service account (in ClusterRoleBinding) as

Kind:User

instead of

Kind : Group.

Example

- kind: User
  name: letsencrypt
  apiGroup: rbac.authorization.k8s.io
-- Bal Chua
Source: StackOverflow