I am using the monitoring app provided by rancher 2.5, which practically is installing the kube-prometheus-stack Helm chart. I am trying to send alerts using a webhook as the receiver. The webhook is on a custom service which logs that it responded to the request with http 400. My question is how can i see the request alertmanager sent to the server that provides the webhook. The i am running the pod with log.level=debug
but the only log i see is:
level=debug ts=2021-01-11T17:22:32.629Z caller=notify.go:685 component=dispatcher receiver=prometheus integration=webhook[0] msg="Notify success" attempts=1
level=debug ts=2021-01-11T17:23:02.591Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=Watchdog[7109567][active]
level=debug ts=2021-01-11T17:23:32.594Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}/{severity=~\"^(?:warning|none|critical)$\"}:{}" msg=flushing alerts=[Watchdog[7109567][active]]
This is not really helpful, i would like to see the actual request and response.
So the way i managed to resolve this was to send the webhook to an intermediate server that would practically run netcat -l
and then i reproduced the request to the actual destination.
A k8s only alternative would be to run: https://github.com/eldadru/ksniff
Sadly i did not find a better solution.