K
Q

Prometheus alert manager configuration in openshift

5/6/2019

I have i have a requirement where in the customer wants to configure some of the alerts as a mandatory ones. Below are the few parameters which iam looking to know more information about the alerts. I was trying to get the details of these parameters and iam not finding it anywhere.

KubeDeploymentReplicasMismatch
KubeStatefulSetReplicasMismatch
KubeStatefulSetGenerationMismatch
KubeDaemonSetRolloutStuck

Could anyone please letme know where can i find more information regarding these parametsrs?

-- sudhir
grafana
kubernetes
openshift
prometheus

Similar Questions

Kubernetes - validate deployments
how to pass environment variable in kubectl deployment?
Jenkins Deployment on Kubernetes has a problem with logs
Access EKS DNS from worker nodes and other EC2 instances in same VPC
k8s pod keeps OOMkilled often

3 Answers

5/6/2019

These are names of Prometheus alerts. You may look into this repo: https://github.com/coreos/kube-prometheus.

All alerts you provided are listed here: https://github.com/coreos/kube-prometheus/blob/master/manifests/prometheus-rules.yaml

KubeStatefulSetReplicasMismatch - StatefulSet has not matched the expected number of replicas for longer than 15 minutes.

KubeStatefulSetGenerationMismatch - StatefulSet generation does not match, this indicates that the StatefulSet has failed but has not been rolled back.

KubeDeploymentReplicasMismatch - Deployment has not matched the expected number of replicas for longer than an hour.

KubeDaemonSetRolloutStuck - Only {{ $value }}% of the desired Pods of DaemonSet are scheduled and ready.

-- Vasily Angapov
Source: StackOverflow
5/6/2019

As an option, can find a full alert config in a prometheus dashboard, /rules. Or Status->Rules menu. enter image description here

The best reasoning of an alert is at the message itself, and in a runbook ref https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/runbook.md

-- Max Lobur
Source: StackOverflow
5/6/2019

You can verify the configuration into prometheus-k8s-rulefiles-0 of ConfigMap.

FYI,

# oc get cm prometheus-k8s-rulefiles-0 -o yaml -n openshift-monitoring | \
  grep -E 'KubeDeploymentReplicasMismatch|KubeStatefulSetReplicasMismatch|KubeStatefulSetGenerationMismatch|KubeDaemonSetRolloutStuck'
      - alert: KubeDeploymentReplicasMismatch
      - alert: KubeStatefulSetReplicasMismatch
      - alert: KubeStatefulSetGenerationMismatch
      - alert: KubeDaemonSetRolloutStuck
-- Daein Park
Source: StackOverflow