I created a job in Prometheus with kubernetes_sd_configs to discover services automatically without the need to declare them manually (Cadvisor, nodeexporter, KubeStateMetrics). I am using helm chart for Prometheus. I tried this job and it can only discover the services in the cluster itself (same cluster with Prometheus server installed) but not the other clusters.

This is the job that I created:

• job_name: 'kubernetes-nodes-cadvisor-aks-NEWUATBACK'

  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  kubernetes_sd_configs:
    - role: node
  
  relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: ks-ba4cf716.hcp.northeurope.azmk8s.io:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor

Does anybody have an idea about this?