K
Q

Kubernetes on docker creates containers with empty serviceaccount and no tokens leading to container crash and restarts

7/9/2016

Still experiencing a similar issues 1.3.0 and upto 1.4.0-alpha.0

In my case (docker based set up), either trusty or kubedns would get unauthorized from api server.

and strangely I see that the secrets are not there inside the instances, under the path /var/run/secrets/kubernetes.io/serviceaccount

[root@ ... ]#  kubectl exec -it kube-dns-v13-htfjo ls /bin/sh
/ #
/ # ls /var/run/secrets/kubernetes.io/serviceaccount
/ #

While it seems they are in the node and in the proxy instance

tmpfs on /var/lib/kubelet/pods/3de53b0c-45bb-11e6-9f03-08002776167a/volumes/kubernetes.io~secret/default-token-8axd8 type
tmpfs on /var/lib/kubelet/pods/3de5591e-45bb-11e6-9f03-08002776167a/volumes/kubernetes.io~secret/default-token-8axd8 type
tmpfs on /var/lib/kubelet/pods/f29f35c7-45cc-11e6-9f03-08002776167a/volumes/kubernetes.io~secret/default-token-ql88q type
  • Deleting the secret and deleting the pods then recreating them has no effect
  • Restarting cluster after unmounting & deleting the folders has no effect either

Naturally this results in kubedns unable to start. Log below

I0709 09:04:11.578816       1 dns.go:394] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0709 09:04:11.578873       1 dns.go:427] records:[], retval:[], path:[local cluster svc default kubernetes]
I0709 09:04:11.579657       1 dns.go:394] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0709 09:04:11.579677       1 dns.go:427] records:[], retval:[], path:[local cluster svc default kubernetes]
E0709 09:04:11.786646       1 reflector.go:216] pkg/dns/dns.go:128: Failed to list *api.Service: serializer for text/html; charset=utf-8 doesn't exist
E0709 09:04:11.786995       1 reflector.go:216] pkg/dns/dns.go:127: Failed to list *api.Endpoints: serializer for text/html; charset=utf-8 doesn't exist
I0709 09:04:12.488674       1 dns.go:145] Ignoring error while waiting for service default/kubernetes: serializer for text/html; charset=utf-8 doesn't exist. Sleeping 1s before retrying.
E0709 09:04:12.879701       1 reflector.go:216] pkg/dns/dns.go:128: Failed to list *api.Service: serializer for text/html; charset=utf-8 doesn't exist
E0709 09:04:12.880000       1 reflector.go:216] pkg/dns/dns.go:127: Failed to list *api.Endpoints: serializer for text/html; charset=utf-8 doesn't exist
I0709 09:04:13.582561       1 dns.go:145] Ignoring error while waiting for service default/kubernetes: serializer for text/html; charset=utf-8 doesn't exist. Sleeping 1s before retrying.
-- Santanu Dey
crash
docker
kube-dns
kubernetes
unauthorized

Similar Questions

How to publish kubernetes LoadBalancer Ingress URL to aws route53
Kubernetes pod stuck in waiting state
Groovy script to apply kubernetes cloud config in jenkins
Kubernetes ui not showing deployed services or deployyment

1 Answer

7/17/2016

This one seems to be a bug still open

https://github.com/kubernetes/kubernetes/issues/26943

The workaround that works is to add rslave option in the kubelet mount like --volume=/var/lib/kubelet:/var/lib/kubelet:rw,rslave as shown below.

This solution also is platform dependent. Read the notes in the bug report.

## Start kubernetes master
sudo docker run \
--volume=/:/rootfs:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:rw \
**--volume=/var/lib/kubelet:/var/lib/kubelet:rw,rslave** \
--volume=/var/run:/var/run:rw \
--net=host \
--privileged=true \
--pid=host \
-d \
gcr.io/google_containers/hyperkube-amd64:${K8S_VERSION} \
/hyperkube kubelet \
    --allow-privileged=true \
    --api-servers=http://localhost:8080 \
    --v=2 \
    --address=0.0.0.0 \
    --enable-server \
    --hostname-override=127.0.0.1 \
    --config=/etc/kubernetes/manifests-multi \
    --containerized \
    --cluster-dns=10.0.0.10 \
    --cluster-domain=cluster.local
-- Santanu Dey
Source: StackOverflow