K
Q

kubernetes api servers showing down in prometheus (kube-state-metrics) --"forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",

10/29/2019

I'm new to k8s, prometheus. I'm trying to collect the metrics of each pods with prometheus but unable to so because of the error: API ERROR.

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}
-- sindhur
kubectl
kubernetes
prometheus

Similar Questions

Where is the root list of CRDs that `operator-courier verify` wants me to update?
add alertmanager to istio prometheus
enable https on local domain with Kubernetes / Traefik Ingress
error: error parsing ssl2.yaml: error converting YAML to JSON: yaml: line 22: did not find expected key

2 Answers

10/31/2019

Create the following manifests:

ServiceAccount.yaml:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
        labels:
            app.kubernetes.io/name: kube-state-metrics
        name: kube-state-metrics
        namespace: grafana

ClusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    labels:
        app.kubernetes.io/name: kube-state-metrics
    name: kube-state-metrics
rules:
    - apiGroups:
        - ""
      resources:
          - configmaps
          - secrets
          - nodes
          - pods
          - services
          - resourcequotas
          - replicationcontrollers
          - limitranges
          - persistentvolumeclaims
          - persistentvolumes
          - namespaces
          - endpoints
      verbs:
          - list
          - watch
    - apiGroups:
        - extensions
      resources:
          - daemonsets
          - deployments
          - replicasets
          - ingresses
      verbs:
          - list
          - watch
    - apiGroups:
        - apps
      resources:
          - statefulsets
          - daemonsets
          - deployments
          - replicasets
      verbs:
          - list
          - watch
    - apiGroups:
        - batch
      resources:
          - cronjobs
          - jobs
      verbs:
          - list
          - watch
    - apiGroups:
        - autoscaling
      resources:
          - horizontalpodautoscalers
      verbs:
          - list
          - watch
    - apiGroups:
        - authentication.k8s.io
      resources:
          - tokenreviews
      verbs:
        - create
    - apiGroups:
        - authorization.k8s.io
      resources:
          - subjectaccessreviews
      verbs:
          - create
    - apiGroups:
        - policy
      resources:
          - poddisruptionbudgets
      verbs:
          - list
          - watch
    - apiGroups:
        - certificates.k8s.io
      resources:
          - certificatesigningrequests
      verbs:
          - list
          - watch
    - apiGroups:
        - storage.k8s.io
      resources:
          - storageclasses
      verbs:
          - list
          - watch
    - nonResourceURLs:
          - "/metrics"
      verbs:
          - get

ClusterRoleBinding.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
    labels:
        app.kubernetes.io/name: kube-state-metrics
    name: kube-state-metrics
roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: kube-state-metrics
subjects:
    - kind: ServiceAccount
      name: kube-state-metrics
      namespace: grafana

And inform your Kube-State-Metrics deployment to use the new ServiceAccount with the following addition to your Template Spec: serviceAccountName: kube-state-metrics.

-- TJ Zimmerman
Source: StackOverflow
10/29/2019

system:anonymous means that an unauthenticated user is trying to get a resource from your cluster, which is forbidden. You will need to create a service account, then give that service account some permissions through RBAC, then make that service account to get the metrics. All that is documented.

As a workaround, you can do this:

kubectl create clusterrolebinding prometheus-admin --clusterrole cluster-admin --user system:anonymous

Now, note that this is a terrible idea, unless you are playing with kubernetes. With this permission you are giving any unauthenticated user total permissions into your cluster.

-- suren
Source: StackOverflow