K
Q

Question

Out of order sample prometheus and cadvisor

6/27/2021

Here is my configmap configuration for prometheus inside my kube cluster.

scrape_configs:
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
    - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
      action: keep
      regex: default;kubernetes;https
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'kubernetes-nodes'
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
    - role: node
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
  - job_name: kube-state-metrics
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    follow_redirects: true
    static_configs:
    - targets:
      - kube-state-metrics.kube-system.svc.cluster.local:8080
  - job_name: kubernetes-cadvisor
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics/cadvisor
    scheme: https
    authorization:
      type: Bearer
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    follow_redirects: true
    relabel_configs:
    - separator: ;
      regex: __meta_kubernetes_node_label_(.+)
      replacement: $1
      action: labelmap
    kubernetes_sd_configs:
    - role: node
      follow_redirects: true
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

In top of the cluster I have a Prometheus federation that federate the in-cluster prometheus.

All things OK, but inside in-cluster prometheus, I have this logs (debug level turned on)

Extracts:

level=debug ts=2021-06-27T11:09:32.130Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/dev/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.130Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.130Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/36edd81cdc0bf2f5213054cf0ee4b6bc86328ec4473b879e3049ee0113a32728/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.130Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5918a77ba85e2430cc0f434cde296c80f3f21f25739f73a9a7cf4296c0b2ad4d/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.130Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5fa3d0a4389e13f8a96a8ff74e22172dd6ffb5c92e5e692a17e3a346660b49c5/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/bc60ea15258c46d3c4cca2e9b28ed608ca89b26ce5b14f2bdb6313d87d762e3b/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/bcb7f220797f48216cb0017b0cce1398ef0a9d377f66fa1f8a742241f9133567/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc4d8fe8e9051bfddc228dda7d272be7514c9cb93f4d1b2d98c9d632c63dfc8a/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/d9033c81c8cb4f0419b4f5ac7f1c14e0d1bb706820f46dc5a98b9db7944b2b08/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/lock\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/user/0\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/run/user/1001\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/sys/fs/cgroup\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.131Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"/var/lib/kubelet/pods/01a67077-3639-41e4-9708-0a3bf1fe5acf/volumes/kubernetes.io~secret/flannel-token-ts79t\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.133Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"overlay_0-48\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.133Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"overlay_0-53\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.133Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"overlay_0-56\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.133Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"overlay_0-62\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.133Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"overlay_0-68\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.133Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"overlay_0-79\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.133Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_current{container=\"\",device=\"overlay_0-93\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.134Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/dev/mapper/debian--vg-root\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.134Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/dev/sda1\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.134Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/dev/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.134Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.134Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/36edd81cdc0bf2f5213054cf0ee4b6bc86328ec4473b879e3049ee0113a32728/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.134Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5918a77ba85e2430cc0f434cde296c80f3f21f25739f73a9a7cf4296c0b2ad4d/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.134Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5fa3d0a4389e13f8a96a8ff74e22172dd6ffb5c92e5e692a17e3a346660b49c5/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/bc60ea15258c46d3c4cca2e9b28ed608ca89b26ce5b14f2bdb6313d87d762e3b/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/bcb7f220797f48216cb0017b0cce1398ef0a9d377f66fa1f8a742241f9133567/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc4d8fe8e9051bfddc228dda7d272be7514c9cb93f4d1b2d98c9d632c63dfc8a/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/containerd/io.containerd.grpc.v1.cri/sandboxes/d9033c81c8cb4f0419b4f5ac7f1c14e0d1bb706820f46dc5a98b9db7944b2b08/shm\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/lock\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/user/0\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/run/user/1001\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/sys/fs/cgroup\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.135Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/var/lib/kubelet/pods/01a67077-3639-41e4-9708-0a3bf1fe5acf/volumes/kubernetes.io~secret/flannel-token-ts79t\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/var/lib/kubelet/pods/0c3c9be9-cc89-4e6e-93f4-e87c9356ad42/volumes/kubernetes.io~secret/kube-proxy-token-59vcr\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/var/lib/kubelet/pods/c05ffb01-231a-43d4-9941-e959ba521f52/volumes/kubernetes.io~secret/x509-certificate-exporter-node-token-m6w4w\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"/var/lib/kubelet/pods/d57f59ba-6ef6-4cd5-84cf-c1e3a2f79433/volumes/kubernetes.io~secret/default-token-zwtjf\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-115\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-121\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-145\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-151\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.136Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-157\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.137Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-164\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.137Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-165\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.137Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-188\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.137Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-44\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.137Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-48\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.137Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-53\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=debug ts=2021-06-27T11:09:32.137Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_fs_io_time_seconds_total{container=\"\",device=\"overlay_0-56\",id=\"/\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}"
level=warn ts=2021-06-27T11:09:32.149Z caller=scrape.go:1467 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Error on ingesting out-of-order samples" num_dropped=303
level=debug ts=2021-06-27T11:09:47.098Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_cpu_load_average_10s{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4fae43df4163b63617776dc1321fe0.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"kube-controller-manager-master2\"}"
level=debug ts=2021-06-27T11:09:47.098Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_cpu_load_average_10s{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a02ba4a6b5572917c3b834d347981b.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"etcd-master2\"}"
level=debug ts=2021-06-27T11:09:47.099Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_cpu_system_seconds_total{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4fae43df4163b63617776dc1321fe0.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"kube-controller-manager-master2\"}"
level=debug ts=2021-06-27T11:09:47.099Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_cpu_system_seconds_total{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a02ba4a6b5572917c3b834d347981b.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"etcd-master2\"}"
level=debug ts=2021-06-27T11:09:47.099Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_cpu_user_seconds_total{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4fae43df4163b63617776dc1321fe0.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"kube-controller-manager-master2\"}"
level=debug ts=2021-06-27T11:09:47.100Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_cpu_user_seconds_total{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a02ba4a6b5572917c3b834d347981b.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"etcd-master2\"}"
level=debug ts=2021-06-27T11:09:47.100Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_file_descriptors{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4fae43df4163b63617776dc1321fe0.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"kube-controller-manager-master2\"}"
level=debug ts=2021-06-27T11:09:47.100Z caller=scrape.go:1511 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://10.10.10.61:10250/metrics/cadvisor msg="Out of order sample" series="container_file_descriptors{container=\"\",id=\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a02ba4a6b5572917c3b834d347981b.slice\",image=\"\",name=\"\",namespace=\"kube-system\",pod=\"etcd-master2\"}"

I suspect Cadvisor and duplicates metrics but I didn't see them.

Kubernetes: 1.20 Prometheus: 2.27.1

-- Achraf Bentabib

cadvisor

kubernetes

prometheus

1 Answer

6/27/2021

You could try by adding some arbitrary label to those metrics, distinguishing the node they belong to:

- job_name: kubernetes-cadvisor
  [...]
  relabel_configs:
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - action: replace
    source_labels: [__meta_kubernetes_node_name]
    target_label: node_name
  [...]

I doubt it would fix them all, but that should help with those related to devices such as /run/lock, /run/user/0, /dev/shm, /dev/sda1, /dev/mapper/debian.+, overlay00-9+, ... as we're likely to find these on all your nodes.

Having done so, let us know which ones are still showing.

-- SYN

Source: StackOverflow

KQ

Out of order sample prometheus and cadvisor

Similar Questions

1 Answer

K
Q