I have a VM on which k8s v1.16.2 is deployed and on top of it prometheus and grafana services running. On grafana dashboard i am able to see all containers and pods performance metrics but for system services only docker and kubelet, Why its not showing other services metrics which are running directly on machine.

Is there any configurations by default kubelet sets to cadvisor which restricts collecting system service metrics..? because with kubelet version v1.13.1 i was able to see system services metrics.

How can i get all service metrics which runs under system.slice through cadvisor of kubelet..?

Verified through :

• Hitting directly Prometheus endpoints.
• Tried directly cadvisor endpoint(k8s exposed endpoint) : /api/v1/nodes/HOSTNAME/proxy/metrics/cadvisor

on both endpoints results i am not seeing any system service metrics except docker and kubelet

Version:

• Docker: 18.09.6,
• Kubelet: v1.16.2,
• CGroup Driver: systemd

Below is Prometheus Configuration:

  metric_relabel_configs:
  - source_labels: [id]
    separator: ;
    regex: ^/machine\.slice/machine-rkt\\x2d([^\\]+)\\.+/([^/]+)\.service$
    target_label: rkt_container_name
    replacement: ${2}-${1}
    action: replace
  - source_labels: [id]
    separator: ;
    regex: ^/system\.slice/(.+)\.service$
    target_label: systemd_service_name
    replacement: ${1}
    action: replace