I have set up a Grafana dashboard, to monitor Kubernetes. It's working as expected but there's an issue of losing metrics/history at some point of time. See the attached screenshot. In the screenshot, no metrics are shown between 21:00 PM to 23:00 PM.
From the above image, is that behavior normal? Thanks.
I'm using Grafana as an addon on Istio.
Part of the Istio helm chart configuration:
grafana:
enabled: true
persist: true
accessMode: ReadWriteManyMy expectation is that no history/metrics should be lost at any specific time.
Istio uses Prometheus to collect the metrics. What you want to do, is to change a Prometheus data retention.
For example, if you are installing it with Helm, just change this parameter in the chart
--set prometheus.retention=2d