printfs hangs in kubernetes environment

8/1/2019

I am debugging an app for a hang. This is running as a container in the kubernetes environment. The app is multi threaded and implements a fuse file system.

 Thread 7 (Thread 0x7f631da4d700 (LWP 20)):
    #0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
    #1  0x00007f632ef80810 in buffered_vfprintf (s=s@entry=0x7f632f30e680 <_IO_2_1_stderr_>, format=format@entry=0x55667b5c4ebe "%s %s at %s:%d ", args=args@entry=0x7f631da4c800) at vfprintf.c:2336
    #2  0x00007f632ef7d726 in _IO_vfprintf_internal (s=0x7f632f30e680 <_IO_2_1_stderr_>, format=0x55667b5c4ebe "%s %s at %s:%d ", ap=ap@entry=0x7f631da4c800) at vfprintf.c:1301
    #3  0x00007f632ef86e54 in __fprintf (stream=<optimized out>, format=<optimized out>) at fprintf.c:32
    ---Type <return> to continue, or q <return> to quit---


other threads also hung in the same place


Thread 17 (Thread 0x7f62bafcd700 (LWP 208)):
#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007f632ef80810 in buffered_vfprintf (s=s@entry=0x7f632f30e680 <_IO_2_1_stderr_>, format=format@entry=0x55667b5c4ebe "%s %s at %s:%d ", args=args@entry=0x7f62bafcb930) at vfprintf.c:2336
#2  0x00007f632ef7d726 in _IO_vfprintf_internal (s=0x7f632f30e680 <_IO_2_1_stderr_>, format=0x55667b5c4ebe "%s %s at %s:%d ", ap=ap@entry=0x7f62bafcb930) at vfprintf.c:1301
#3  0x00007f632ef86e54 in __fprintf (stream=<optimized out>, format=<optimized out>) at fprintf.c:32
}

Thread 15 (Thread 0x7f62c47e0700 (LWP 151)):
---Type <return> to continue, or q <return> to quit---
{
#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007f632ef80810 in buffered_vfprintf (s=s@entry=0x7f632f30e680 <_IO_2_1_stderr_>, format=format@entry=0x55667b5c4ebe "%s %s at %s:%d ", args=args@entry=0x7f62c47de930) at vfprintf.c:2336
#2  0x00007f632ef7d726 in _IO_vfprintf_internal (s=0x7f632f30e680 <_IO_2_1_stderr_>, format=0x55667b5c4ebe "%s %s at %s:%d ", ap=ap@entry=0x7f62c47de930) at vfprintf.c:1301
#3  0x00007f632ef86e54 in __fprintf (stream=<optimized out>, format=<optimized out>) at fprintf.c:32

I dont know much about the bugs in the kubernetes logging infrastucture . The code doesnt hang when run outside kubernetes though.

-- eric johnson
docker
kubernetes
libc
multithreading
printf

0 Answers