Kubernetes 401 Unauthorized, token not copied to containers

12/28/2016

Service Accounts are being created with secrets, however upon pod creation it looks like the token and ca cert are not copied into the pod.

Logs show:

[2016-12-27 16:16:29,012][WARN ][io.fabric8.elasticsearch.discovery.k8s.K8sUnicastHostsProvider] 
[James Dr. Power] Exception caught during discovery javax.ws.rs.WebApplicationException : HTTP 401 Unauthorized

Environment variables of the pod include:

KUBERNETES_CA_CERTIFICATE_FILE=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

However, the directory /var/run is empty

es-master-gqq7m:/var/run# pwd
/var/run 

es-master-gqq7m:/var/run# ls -la 
total 0
drwxr-xr-x    2 root     root             6 Jun 12  2015 . 
drwxr-xr-x    9 root     root            81 Jun 12  2015 ..

The full definition of the replication controller is here, and includes:

spec:
  serviceAccount: elasticsearch
  env: 
    - name: KUBERNETES_CA_CERTIFICATE_FILE 
      value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

[cstelly@smcluster ~]$ kubectl get serviceaccounts
NAME            SECRETS   AGE
default         1         29d
elasticsearch   1         4d

Any ideas?

Thanks!


Edit

In response to this, currently the apiconfig has the flag

--admission-control=AlwaysAdmit \

As for volume info, I ran

kubectl describe pod <pod-name>, and the only volume information I see is:

Volumes:
  storage:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
-- Rubber Duck
distributed-computing
kubernetes

2 Answers

12/29/2016

The service account token secret should be added as a volume when the pods are created. This is done by the ServiceAccount admission plugin.

Some questions:

  1. If you inspect one of the running pods in the API, does it include a volume and volume mount referencing a service account token?

  2. What admission plugins do you have configured for your API server?

-- Jordan Liggitt
Source: StackOverflow

12/30/2016

The token is itself created during admission control processing.

The setting recommended for >1.4 is

--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota

Mind the "ServiceAccount" part!

See original doc

-- Maxym
Source: StackOverflow