In my Kubernetes cluster, I have Prometheus, Grafana in monitoring stack and EFK stack for logs.
I created some Grafana alerts fired by the metrics coming from node-exporter.
Also, I'm able to see the Kubernetes node (VM) logs on Kibana.
I wanna create alerts on Grafana when a node has no logs for some time.
What is the best way to do that?
I connected ElasticSearch (ES) to Grafana as a data source. I'm able to see ES log metrics on a Grafana chart. But, this solution seems problematic.
Because the cluster might be - down-scaled - upgraded, when all old nodes are gone and new nodes are created.
The first is not a very big deal (if the alert is fired only on meeting the condition the first time)
The second might cause dozens of alerts.
You need to monitor node resource consumption to ensure all nodes in your cluster are healthy. Use following data: enough nodes in your cluster, resource allocation is sufficient for deployed applications, etcd is healthy, you're not hitting any resources.
NewRelic solution can helps you with that, it tracks resource consumption (used cores and memory) for each Kubernetes node. That lets you track the number of network requests sent across containers on different nodes within a distributed service.
If you set alerts, you’ll be notified if node stop reporting (has no logs) or if a node’s CPU or memory usage drops below a desired threshold.