How to view C++ application Core Dumps generated inside Docker Containers through FluentD for GCE

2/19/2018

I'm running a c++ application inside my Kubernetes Cluster. With the help of GCE StackDriver and fluentd-gcp, I'm able to see my application logs. Now, I would like to see code dumps (generated if my app crashes) through GCE StackDriver interface .. Is it possible to achieve this using FluentD ?? If yes, any recommended FluentD plug-ins that does the job ..

Thanks,

-- bms
docker
fluentd
google-cloud-stackdriver
kubernetes
stackdriver

1 Answer

8/4/2018

I was recently thinking about similar problem - how to get backtrace of crashed c++ app running in a cluster which I don't have acces to.

I have come up with an idea to run/wrap my app with a script like this:

#!/bin/bash

function coredumps_monitor
{
    EXECUTABLE=$1
    COREDIR=$2

    while true; do
        sleep 2

        for f in $(find $COREDIR -type f -name 'core*'); do
            sleep 2

            BT=$(gdb -batch -ex 'thread apply all bt' $EXECUTABLE $f)

            jq -nc \
                --arg bt "${BT}" \
                --arg file "${f}" \
                '{event: "coredump", bt: $bt, file: $file}'

            mv $f latest-core
        done
    done
}

coredumps_monitor /my-cpp-server /dir-with-coredumps &

exec /my-cpp-server

It's necassary to install 'gdb-minimal' and 'jq' packages, which was just additional 8 MB in my case.

As you can see the idea is to run a process (alongside with my app) which periodically checks for any coredums. For the coredumps found it gets the backtrace with 'gdb' and print it to stdout in json format created with 'jq' tool.

Our server has forking model, so it's very unlikely that the server itself and hence the container crashes. But if that happens anyway it's possible to store coredumps in a volume and print backtraces when the container is re-created again.

There might be a better solution, but I haven't come across any so far. Also note that I'm still new to clound :) and we don't use GCE.

-- VladimĂ­r Burian
Source: StackOverflow