Error Prometheus endpoint for checking AlertManager
I installed Prometheus (follow in this link: https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/)
But, when checking status of Targets, it shows "Down" for AlertManager service, every another endpoint are up, please see the attached file 
Then, I check Service Discovery, the discovered labels shows:
"address\="192.168.180.254:9093" __meta_kubernetes_endpoint_address_target_kind="Pod" __meta_kubernetes_endpoint_address_target_name="alertmanager-6c666985cc-54rjm" __meta_kubernetes_endpoint_node_name="worker-node1" __meta_kubernetes_endpoint_port_protocol="TCP" __meta_kubernetes_endpoint_ready="true" __meta_kubernetes_endpoints_name="alertmanager" __meta_kubernetes_namespace="monitoring" __meta_kubernetes_pod_annotation_cni_projectcalico_org_podIP="192.168.180.254/32" __meta_kubernetes_pod_annotationpresent_cni_projectcalico_org_podIP="true" __meta_kubernetes_pod_container_name="alertmanager" __meta_kubernetes_pod_container_port_name="alertmanager" __meta_kubernetes_pod_container_port_number="9093""
But Target Labels show another port (8080), I don't know why:
instance="192.168.180.254:8080"
job="kubernetes-service-endpoints"
kubernetes_name="alertmanager"
kubernetes_namespace="monitoring"
Similar Questions
2 Answers
I've reproduced your issue on GCE.
If you are using version 1.16+ you have probably changed apiVersion as in tutorial you have Deployment in extensions/v1beta1. Since K8s 1.16+ you need to change it to apiVersion: apps/v1. Otherwise you will get error like:
error: unable to recognize "STDIN": no matches for kind "Deployment" in version "extensions/v1beta1"Second thing, in 1.16+ you need to specify selector. If you will not do it you will receive another error:
`error: error validating "STDIN": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false`It would look like:
...
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-server
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
...Regarding port 8080 please check this article with example.
Port: Port is the port number which makes a service visible to other services running within the same K8s cluster. In other words, in case a service wants to invoke another service running within the same Kubernetes cluster, it will be able to do so using port specified against “port” in the service spec file.
It worked for my environment in GCE. Did you configure firewall for your endpoints?
In addition. In Helm 3 some hooks were deprecated. You can find this information here.
If you still have issue please provide your YAMLs witch applied changes to version 1.16+.
First, if you want to install prometheus and grafana without getting sick, you need to do it though helm.
And then
helm install installationWhatEverName stable/prometheus-operator