I have an application running on kubernetes (It is a cluster running on cloud) and want to setup monitoring and logging for that application. There are various possibilities for the setup. What would be the best practices of doing that, like recommended method or industrial standard?
Can anyone please help me with some suggestions regarding best practices?
It all depends on how many clusters you have. If you have one cluster, the application you want to monitor on it will be the best choice option 1:
- A prometheus monitoring setup inside kubernetes cluster : prometheus-operator helm chart installed inside the cluster that can monitor the entire cluster, including the application.
The advantages of such a solution include possibly simple and quick configuration, in addition, you have everything in one place (application and Prometheus) and you do not need a new cluster to monitor another. Here you can find example tutorial.
However, if you plan to expand to many clusters, or you already need to monitor many clusters, option 3 will be the best choice:
- A prometheus federation setup where one external prometheus setup gets metrics from an internal prometheus setup of k8s.
Thanks to this solution, you will have all the metrics in one place, regardless of the number of clusters you need to monitor:
Commonly, it is used to either achieve scalable Prometheus monitoring setups or to pull related metrics from one service's Prometheus into another.
You can find example tutorials about Prometheus federation in Kubernetes and Monitoring multiple federated clusters with Prometheus - the secure way