Measuring the CPU metrics over Kubernetes Container had me puzzle with a few thing (listed below)

PROMQL query and its result

container_cpu_user_seconds_total{namespace="app",pod=~"level3-parser.*"} + container_cpu_system_seconds_total{namespace="app",pod=~"level3-parser.*"}

{container="level3-parser", endpoint="https-metrics", id="/kubepods/besteffort/pod28c71245-666e-41be-8596-72bde247fe0b/4322ec7936791f93f7ad2cbcc5af6e8d84841157d39a26c523cbc220e491db57", image="cdrarchiverregistry.azurecr.io/cdr_parser:0.16.0", instance="172.19.51.138:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="4322ec7936791f93f7ad2cbcc5af6e8d84841157d39a26c523cbc220e491db57", namespace="app", node="aks-systempool-38751455-vmss000000", pod="level3-parser-78b4c89547-vdv86", service="monitoring-prometheus-oper-kubelet"}
129.05
{endpoint="https-metrics", id="/kubepods/besteffort/pod28c71245-666e-41be-8596-72bde247fe0b", instance="172.19.51.138:10250", job="kubelet", metrics_path="/metrics/cadvisor", namespace="app", node="aks-systempool-38751455-vmss000000", pod="level3-parser-78b4c89547-vdv86", service="monitoring-prometheus-oper-kubelet"}
129.18
{endpoint="https-metrics", id="/kubepods/besteffort/pod28c71245-666e-41be-8596-72bde247fe0b/cc9d345ac64d8b0ec1ea71602d46b1448706a8b32426022c12fb6e28ca1a3267", image="mcr.microsoft.com/oss/kubernetes/pause:3.5", instance="172.19.51.138:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="cc9d345ac64d8b0ec1ea71602d46b1448706a8b32426022c12fb6e28ca1a3267", namespace="app", node="aks-systempool-38751455-vmss000000", pod="level3-parser-78b4c89547-vdv86", service="monitoring-prometheus-oper-kubelet"}
0.02

container_cpu_system_seconds_total{namespace="app",pod=~"level3-parser.*"}

container_cpu_system_seconds_total{container="level3-parser", endpoint="https-metrics", id="/kubepods/besteffort/pod28c71245-666e-41be-8596-72bde247fe0b/4322ec7936791f93f7ad2cbcc5af6e8d84841157d39a26c523cbc220e491db57", image="cdrarchiverregistry.azurecr.io/cdr_parser:0.16.0", instance="172.19.51.138:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="4322ec7936791f93f7ad2cbcc5af6e8d84841157d39a26c523cbc220e491db57", namespace="app", node="aks-systempool-38751455-vmss000000", pod="level3-parser-78b4c89547-vdv86", service="monitoring-prometheus-oper-kubelet"}
53.25
container_cpu_system_seconds_total{endpoint="https-metrics", id="/kubepods/besteffort/pod28c71245-666e-41be-8596-72bde247fe0b", instance="172.19.51.138:10250", job="kubelet", metrics_path="/metrics/cadvisor", namespace="app", node="aks-systempool-38751455-vmss000000", pod="level3-parser-78b4c89547-vdv86", service="monitoring-prometheus-oper-kubelet"}
53.34
container_cpu_system_seconds_total{endpoint="https-metrics", id="/kubepods/besteffort/pod28c71245-666e-41be-8596-72bde247fe0b/cc9d345ac64d8b0ec1ea71602d46b1448706a8b32426022c12fb6e28ca1a3267", image="mcr.microsoft.com/oss/kubernetes/pause:3.5", instance="172.19.51.138:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="cc9d345ac64d8b0ec1ea71602d46b1448706a8b32426022c12fb6e28ca1a3267", namespace="app", node="aks-systempool-38751455-vmss000000", pod="level3-parser-78b4c89547-vdv86", service="monitoring-prometheus-oper-kubelet"}
0.01

First, I had an understanding that

TOTAL CPU USAGE  = CPU USER + CPU SYSTEM 

Also mentioned here

But the result from PROMQL does adhere to that.

And Second, there are multiple metrics one with container label set and the other with container label not set what is the difference between them and which one of the two ones should use.