I am trying to get stacktrace of my process which is running inside pod of k8s. For that I have installed gstack binary into it, but it does not give me any output.
Here is the sample demo:
sh-4.3# gstack
Usage: gstack <process-id>
sh-4.3# sleep 100 &
[2] 1594
sh-4.3# gstack 1594
sh-4.3# gstack 1594 > test
sh-4.3# cat test
sh-4.3#Can someone help me on it?
PS: It is fine if you have other way to gather stack trace (other then gstack) too.
You should connect to pod as root to view traces.
Used latest opensuse/tumbleweed for example pod
#suse-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: suse
spec:
containers:
- name: suse
image: opensuse/tumbleweed
command: [ "/bin/bash", "-ce", "tail -f /dev/null" ]Connected to pod in regular way and start the process
suse:/ # kubectl exec suse -ti -- bash
suse:/ # sleep 100000&
[1] 677
suse:/ # ps aux | grep sleep
root 677 0.0 0.0 2324 764 pts/3 S 14:16 0:00 sleep 100000Check:
suse:/ # gstack 677 --> empty
suse:/ # gdb
(gdb) attach 677
Attaching to process 677
ptrace: Operation not permitted.ptrace: Operation not permitted. <-- this is root cause
What you can do:
1) you can(as I did) install 3rd party awesome kubectl-plugins and use kubectl ssh -u root [pod] to access pod under root.
git clone https://github.com/jordanwilson230/kubectl-plugins.git
cd kubectl-plugins
./install-plugins.sh
source ~/.bash_profile
kubectl ssh -u root suse
Connecting...
Pod: suse
Namespace: NONE
User: root
Container: NONE
Command: /bin/sh
If you don't see a command prompt, try pressing enter.
sh-5.0# gstack 677
#0 0x00007f8eeb2a7603 in nanosleep () from /lib64/libc.so.6
#1 0x000055fd4fc88677 in ?? ()
#2 0x000055fd4fc88450 in ?? ()
#3 0x000055fd4fc85500 in ?? ()
#4 0x00007f8eeb201e0b in __libc_start_main () from /lib64/libc.so.6
#5 0x000055fd4fc855da in ?? ()
sh-5.0# gdb
GNU gdb (GDB; openSUSE Tumbleweed) 8.3.1
(gdb) attach 677
Attaching to process 677
Reading symbols from /usr/bin/sleep...
(No debugging symbols found in /usr/bin/sleep)
Reading symbols from /lib64/libc.so.6...
(No debugging symbols found in /lib64/libc.so.6)
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)
0x00007f8eeb2a7603 in nanosleep () from /lib64/libc.so.6
Missing separate debuginfos, use: zypper install coreutils-debuginfo-8.31-2.2.x86_642) You can follow @mac answer
-find out what node it is running on kubectl get po -n [NAMESPACE] -o wide
-ssh node
-find the docker container sudo docker ps | grep [namespace]
-log into container as root sudo docker exec -it -u root [DOCKER ID] /bin/bash
Hope it helps