recently I've been unable to connect to the bash on my kubernetes cluster. And I'm at a loss as to why. Have anyone else experienced this?
What happened: I can no longer connect to /bin/bash in my running pods. It simply hangs when trying to exec the command. I've verified that bash is installed (/bin/bash --version). I've tried both locally and from the google cloud console
What you expected to happen: That my local terminal successfully connects to the pods bash terminal
How to reproduce it (as minimally and precisely as possible): I've only tested it on my cluster, but the command I'm running is:
kubectl exec -i POD_ID --namespace=NAMESPACE -c CONTAINER -- /bin/bash
I've also run it with DEBUG=1 which results in the following:
DEBUG=1 kubectl exec -i POD_ID --namespace=NAMESPACE -c CONTAINER --
/bin/bash
I0412 10:52:14.560443 2675 logs.go:41] (0xc4200aed10)
(0xc420242140) Create stream
I0412 10:52:14.560486 2675 logs.go:41] (0xc4200aed10)
(0xc420242140) Stream added, broadcasting: 1
I0412 10:52:14.611561 2675 logs.go:41] (0xc4200aed10) Reply frame
received for 1
I0412 10:52:14.611658 2675 logs.go:41] (0xc4200aed10)
(0xc4203e26e0) Create stream
I0412 10:52:14.611692 2675 logs.go:41] (0xc4200aed10)
(0xc4203e26e0) Stream added, broadcasting: 3
I0412 10:52:14.656684 2675 logs.go:41] (0xc4200aed10) Reply frame
received for 3
I0412 10:52:14.656725 2675 logs.go:41] (0xc4200aed10)
(0xc4202425a0) Create stream
I0412 10:52:14.656737 2675 logs.go:41] (0xc4200aed10)
(0xc4202425a0) Stream added, broadcasting: 5
I0412 10:52:14.702100 2675 logs.go:41] (0xc4200aed10) Reply frame
received for 5
I0412 10:52:14.702140 2675 logs.go:41] (0xc4200aed10)
(0xc420659680) Create stream
I0412 10:52:14.702151 2675 logs.go:41] (0xc4200aed10)
(0xc420659680) Stream added, broadcasting: 7
I0412 10:52:14.746707 2675 logs.go:41] (0xc4200aed10) Reply frame
received for 7Anything else we need to know?:
Environment: - Kubernetes version
Client Version: version.Info{Major:"1", Minor:"8",
GitVersion:"v1.8.6",
GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c",
GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z",
GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+",
GitVersion:"v1.9.6-gke.0",
GitCommit:"cb151369f60073317da686a6ce7de36abe2bda8d",
GitTreeState:"clean", BuildDate:"2018-03-21T19:01:20Z",
GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}Google CloudOS (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
Kernel (e.g. uname -a): Linux bounce-deployment-f95687cbc-mfgtg 4.4.111+ #1 SMP Thu Feb 1 22:06:37 PST 2018 x86_64 GNU/Linux
Install tools:
It could have been that $DEBUG is conflicting with the k8s's internal debugging flag. Try rename or remove $DEBUG environment variable.
I ran into a similar issue - when you set environment variable DEBUG to truthy values, the kubernetes container went nuts with all sorts of logs.go print out all over the screen with every single key stroke (just like what you saw there in the log).
I disabled the environment variable DEBUG by setting it to 0 or false, it went back to normal. kubectl exec works with a normal shell prompt.
You have to pass -t to kubectl exec command, just like docker exec -i -t <container_name> bash.
From the help command:
-t, --tty=false: Stdin is a TTY