Debug alertmanager webhook

1/11/2021

I am using the monitoring app provided by rancher 2.5, which practically is installing the kube-prometheus-stack Helm chart. I am trying to send alerts using a webhook as the receiver. The webhook is on a custom service which logs that it responded to the request with http 400. My question is how can i see the request alertmanager sent to the server that provides the webhook. The i am running the pod with log.level=debug but the only log i see is:

level=debug ts=2021-01-11T17:22:32.629Z caller=notify.go:685 component=dispatcher receiver=prometheus integration=webhook[0] msg="Notify success" attempts=1
level=debug ts=2021-01-11T17:23:02.591Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=Watchdog[7109567][active]
level=debug ts=2021-01-11T17:23:32.594Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}/{severity=~\"^(?:warning|none|critical)$\"}:{}" msg=flushing alerts=[Watchdog[7109567][active]] 

This is not really helpful, i would like to see the actual request and response.

-- Kroustou
kubernetes
prometheus
prometheus-alertmanager
prometheus-operator
webhooks

1 Answer

1/12/2021

So the way i managed to resolve this was to send the webhook to an intermediate server that would practically run netcat -l and then i reproduced the request to the actual destination.

A k8s only alternative would be to run: https://github.com/eldadru/ksniff

Sadly i did not find a better solution.

-- Kroustou
Source: StackOverflow