I set up Grafana to run in GKE (Kubernetes) with a service and default Ingress controller to open it to the internet. Everything is working without any issues.
After creating some dashboards I wanted to setup Slack alerting using the slack webhook. After filling out all the details I received a 502 bad gateway error.
I have setup a second service to open port 443(Default slack webhook port) and exposed it with kubectl expose deployment --type=NodePort --port=443 and have also tried --type=LoadBalancer with no luck.
I've also tried setting up a second Ingress service pointing the second service, but then I run into readinessProbe issues.
Anyone had the same issue and if so how was it resolved?
Network Policy was enabled on the cluster and there were cluster policies denying outgoing traffic. After setting up my network policies for the pods in my own namespace I was able to connect without any issues.