K
Q

serviceAccount Token Volume Projection - Projected Token in "path" in manifest file is incorrect

7/24/2019

I am using serviceAccount Token Volume projection as described here. This is the manifest file I'm using:

kind: Pod
apiVersion: v1
metadata:
  name: pod-acr
spec:
  containers:
  - image: yorchaksacr.azurecr.io/busybox:1.0
    name: busybox
    command: ["/bin/sh","-c","sleep 36000"]
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: vault-token
  serviceAccountName: pull-acr-images
  volumes:
  - name: vault-token
    projected:
      sources:
      - serviceAccountToken:
          path: vault-token
          expirationSeconds: 7200
          audience: vault

As expected, the token is mounted to the container under /var/run/secrets/tokens/vault-token:

/ # ls -la /var/run/secrets/tokens
total 4
drwxrwxrwt    3 root     root           100 Jul 24 21:35 .
drwxr-xr-x    4 root     root          4096 Jul 24 21:35 ..
drwxr-xr-x    2 root     root            60 Jul 24 21:35 ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            31 Jul 24 21:35 ..data -> ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            18 Jul 24 21:35 vault-token -> ..data/vault-token

Problem is if I try to authenticate to the API server using this token the API rejects the call with 401 Unauthorized:

/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/tokens/vault-token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
wget: server returned error: HTTP/1.1 401 Unauthorized

However if I use the default path and token where service account tokens are projected for all pods /var/run/secrets/kubernetes.io/serviceacconts/token that works:

/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
saving to 'index.html'
index.html           100% |************************************************************************************************************************************************************|  2738  0:00:00 ETA
'index.html' saved

If I cat both tokens I can see they are actually different:

# cat /var/run/secrets/tokens/vault-token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJhdWQiOlsidmF1bHQiXSwiZXhwIjoxNTY0MDEzMjcwLCJpYXQiOjE1NjQwMDYwNzAsImlzcyI6Imh0dHBzOi8vMTAuMi4xLjE5Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0IiwicG9kIjp7Im5hbWUiOiJwb2QtYWNyIiwidWlkIjoiNThiNjI5YWEtZGU4Ni00YTAzLWI3YmQtMTI4ZGFiZWVkYmQ5In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJ1aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YifX0sIm5iZiI6MTU2NDAwNjA3MCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6cHVsbC1hY3ItaW1hZ2VzIn0.UjPaTgWHPwCeeh1ltBb64hv0yEjKkRxKw_BZ3PLDA3HsJK-keXN40Khp-8mNnLQ-uYIfMgW4FXwYIm0SVeQUhM4sh4rwjAYDEfEHDah9AvhEL8I65T_jhnhT10E1M7mzk1x0RFGvjZAECd1RlYM7IuXIkEfZCI_6GRVAbX3Vmk6XF0sRh2T8DZzw8kj_Z54J2gYCt2beBnn7hC9rOC9LW9J0AFEAAQQE_UJME5y4jZD6hfJMSGOouyQm70nVGytqKVsLbzyorH5pugEqrs1Z_dLx6E3Ta9kELRPvyDZgeNiS44fEYlRApn6fZawsppc1oRNoeyMqiIPRdgQekBVfTA/ #


# cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InB1bGwtYWNyLWltYWdlcy10b2tlbi1oYjU0NyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwdWxsLWFjci1pbWFnZXMifQ.nqqhZVmBUuKVi6E3L9MEn8oW1dKd-DV4c9jcVy5mXAuEMZ1WgLlaaHFF1ibnVMjEK6VUJyJhp7w08hgSmyyh-KY4BQ5oJf1jmSySvmttJxjXW-KsMpf5rHF0ZDmgaqZwbi7FvowtoTECstFBVNoszKJUn1iV5mU_6MQkEtGTNyE4KuZ9LEvPuZxiNZ5UyW3UaHXLqF63-w_xlkfa_75E-cgXqvSSGTCb6RsTuOmVyCqganx5SpIb5EU-3Mu7hUWEhSRAh3tpcPIwjS7-NkuO0ReH7Z40rPHqkIokshUUO75WM_oPq7tlu6PSCTwOK-Jw66kzi-jqKNyKvMeWJUq4WQ/ #

Does anybody have an idea why am I seeing this behavior? I would expect both tokens to work, but apparently looks like it is not the case.

Configuration of the API Server:

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.2.1.19
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --basic-auth-file=/etc/kubernetes/pki/passwordfile
    - --service-account-issuer=https://10.2.1.19
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    image: k8s.gcr.io/kube-apiserver:v1.15.1
    imagePullPolicy: IfNotPresent
-- Jorge Cortes - MSFT
kubernetes

Similar Questions

How can I set the container order for a multi-container pod in Kubernetes (with Kustomize)?
Cannot access my Kubernetes cluster. Error: You must be logged in to the server (Unauthorized)
How to attach an EKS volume directly to an EKS Pod
Kubernetes operator without CRD

1 Answer

7/25/2019

Confusingly, 401 Unauthorized indicates an authentication problem rather than an authorization problem (see here). This means the Kubernetes service account token authenticator doesn't like the token at /var/run/secrets/tokens/vault-token (let's call it the vault-token). The two tokens differ in a couple ways. Here's the decoded vault-token:

{
  "aud": [
    "vault"
  ],
  "exp": 1564013270,
  "iat": 1564006070,
  "iss": "https://10.2.1.19",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "pod-acr",
      "uid": "58b629aa-de86-4a03-b7bd-128dabeedbd9"
    },
    "serviceaccount": {
      "name": "pull-acr-images",
      "uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f"
    }
  },
  "nbf": 1564006070,
  "sub": "system:serviceaccount:default:pull-acr-images"
}

Notice the audiences (["vault"]), issuer ("https://10.2.1.19"), and subject ("system:serviceaccount:default:pull-acr-images").

Here's the default-path-token:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "pull-acr-images-token-hb547",
  "kubernetes.io/serviceaccount/service-account.name": "pull-acr-images",
  "kubernetes.io/serviceaccount/service-account.uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f",
  "sub": "system:serviceaccount:default:pull-acr-images"
}

Same subject, but different issuer ("kubernetes/serviceaccount") and no audiences.

I'm not sure why the default-path-token has a different issuer, or why it's authenticating correctly, but your vault-token is not authenticating correctly because the audiences don't match the issuer.

More specifically, the clue is in the doc you linked here. It says that this feature working correctly depends on how you set the following flags for the kube-apiserver:

  • --service-account-issuer
  • --service-account-signing-key-file
  • --service-account-api-audiences

Not sure if those docs are wrong or just out of date, but since you're using v1.15.1 the flags are now called:

  • --service-account-issuer
  • --service-account-signing-key-file string
  • --api-audiences

The flag documentation says of the --api-audiences flag:

Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.

Since you don't have this flag set and you have --service-account-issuer=https://10.2.1.19, and you have audience: vault in your pod spec, your token is going to claim its bound to the vault audience, and the token authenticator is going to try to match this against the value of the --service-account-issuer flag, and clearly those don't match.

You can make these match by specifying audience: https://10.2.1.19 in your pod spec instead of audience: vault. One caveat: this solution may technically work to ensure the token authenticates, but I'm not sure what the truly right answer is insofar as using this flags and fields in the pod spec as they're truly intended, and just making these strings match could be a bit hacky.

-- Amit Kumar Gupta
Source: StackOverflow