K
Q

Unable to get a shell into citadel container in kubernetes

11/5/2018

I have Istio (including citadel) running in minikube using the instructions at https://istio.io/docs/setup/kubernetes/helm-install .

$ helm template install/kubernetes/helm/istio --name istio --namespace istio-system > $HOME/istio.yaml
$ kubectl create namespace istio-system
$ kubectl apply -f $HOME/istio.yaml

When I try to get a shell into the citadel container, I am getting an error:

$ kubectl exec -it istio-citadel-6d7f9c545b-bkvnx  -- /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown
command terminated with exit code 126

However, I can exec into other containers like pilot fine.

These are my pods and containers, if it helps.

shell-demo:                             nginx,
istio-citadel-6d7f9c545b-bkvnx:         docker.io/istio/citadel:1.0.3,
istio-cleanup-secrets-rp4wv:            quay.io/coreos/hyperkube:v1.7.6_coreos.0,
istio-egressgateway-866885bb49-6jz9q:   docker.io/istio/proxyv2:1.0.3,
istio-galley-6d74549bb9-7nhcl:          docker.io/istio/galley:1.0.3,
istio-ingressgateway-6c6ffb7dc8-bvp6b:  docker.io/istio/proxyv2:1.0.3,
istio-pilot-685fc95d96-fphc9:           docker.io/istio/pilot:1.0.3, docker.io/istio/proxyv2:1.0.3,
istio-policy-688f99c9c4-bpl9w:          docker.io/istio/mixer:1.0.3, docker.io/istio/proxyv2:1.0.3,
istio-security-post-install-s6dft:      quay.io/coreos/hyperkube:v1.7.6_coreos.0,
istio-sidecar-injector-74855c54b9-6v5xg:docker.io/istio/sidecar_injector:1.0.3,
istio-telemetry-69b794ff59-f7dv4:       docker.io/istio/mixer:1.0.3, docker.io/istio/proxyv2:1.0.3,
prometheus-f556886b8-lhdt8:             docker.io/prom/prometheus:v2.3.1,
coredns-c4cffd6dc-6xblf:                k8s.gcr.io/coredns:1.2.2,
etcd-minikube:                          k8s.gcr.io/etcd-amd64:3.1.12,
kube-addon-manager-minikube:            k8s.gcr.io/kube-addon-manager:v8.6,
kube-apiserver-minikube:                k8s.gcr.io/kube-apiserver-amd64:v1.10.0,
kube-controller-manager-minikube:       k8s.gcr.io/kube-controller-manager-amd64:v1.10.0,
kube-dns-86f4d74b45-bjk54:              k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8, k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8, k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8,
kube-proxy-mqfb9:                       k8s.gcr.io/kube-proxy-amd64:v1.10.0,
kube-scheduler-minikube:                k8s.gcr.io/kube-scheduler-amd64:v1.10.0,
kubernetes-dashboard-6f4cfc5d87-zwk2c:  k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0,
storage-provisioner:                    gcr.io/k8s-minikube/storage-provisioner:v1.8.1,

When I do minikube ssh and then try to exec into the citadel container, I am getting similar error:

$ docker ps | grep citadel
f173453f843c        istio/citadel                              "/usr/local/bin/isti…"   3 hours ago         Up 3 hours                              k8s_citadel_istio-citadel-6d7f9c545b-bkvnx_istio-system_3d7b4f08-e120-11e8-bc40-ee7dbbb8f91b_0
7e96617d81ff        k8s.gcr.io/pause-amd64:3.1                 "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_istio-citadel-6d7f9c545b-bkvnx_istio-system_3d7b4f08-e120-11e8-bc40-ee7dbbb8f91b_0

$ docker exec -it f173453f843c sh
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown

$ docker exec -it f173453f843c /bin/sh
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown

$ docker exec -it f173453f843c ls
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"ls\": executable file not found in $PATH": unknown

I can see the citadel containers logs fine. The logs are available at https://pastebin.com/xTy9vSz2

Do you know why we can't exec into citadel container?

Thanks for reading.

-- user674669
docker
istio
kubernetes
minikube

Similar Questions

How can you use the kubectl tool (in a stateful/local way) for multiple managing multiple clusters from different directories simultaneously?
statefulset stuck on volumeClaimTemplates pending
Committed Usage cost more than non-comitted usage Kubernetes Engine
TCP ingress support in Kubernetes

2 Answers

11/5/2018

You can't shell in because neither sh nor bash are available in the container. A lot of times these are removed for the sake of efficiency and having a minimal container image.

If you'd like to shell into the container I recommend you build your own image in include bash or sh in it.

You can see here that the Dockerfile builds an image that has nothing but the static binary. For that, you want to change the base image. For example:

FROM alpine

instead of:

FROM scratch

Hope it helps.

-- Rico
Source: StackOverflow
7/24/2019

Do you know why we can't exec into citadel container?

Starting with the next Kubernetes release (1.16+, Q3 2019), you can.

See kubernetes/kubernetes PR 59416 (PR="pull request"): "Add Ephemeral Containers to the Kubernetes core API" (commit 7e6b70f).
With PR 79614 for the documentation.

It is made to resolve issue 27140: "Support for troubleshooting distroless containers".

(pnnl-miscscripts/miscscripts/bin is a workaround)

An EphemeralContainer is a temporary container that may be added to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod removed or restarted.

-- VonC
Source: StackOverflow