I installed the stable/prometheus helm chart with some minor changes proposed at helm/charts#17268 to make it compatible with Kubernetes v1.16
After installation, none of the Kubernetes grafana dashboards show correct values. I am using 8769 (https://grafana.com/grafana/dashboards/8769) dashboard which provides many information on cpu, memory, network, etc. This dashboard is working properly on older k8s versions but on v1.16 it shows no results. I also randomly tried some other dashboards (8588, 6879, 10551) but they either just show the requested resource for each pod and not the live usage or showing nothing.
What these dashboards do is they send a promql query to prometheus and get the results. For example this is the promql query for cpu usage from 8769 dashboard:
sum (rate (container_cpu_usage_seconds_total{id!="/",namespace=~"$Namespace",pod_name=~"^$Deployment.*quot;}[1m])) by (pod_name)I don't know if I have to change the promql or the problem is somewhere else.
Try the installation this way, as the new CRDs had some issue, so I used old CRDs-
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/podmonitor.crd.yaml
helm install --name prometheus --namespace monitoring stable/prometheus-operator --set prometheusOperator.createCustomResource=false
Make sure that CRD's don't exist priory, you can delete them via
kubectl delete crd --allKubernetes 1.16 removes the labels pod_name and container_name from cAdvisor metrics, duplicates of pod and container.
You need change pod_name -> pod, container_name -> container in Grafana dashboards JSON models.