Cluster level logging using Elasticsearch and Kibana on Docker for Windows

8/16/2018

The Kubernetes documentation states it's possible to use Elasticsearch and Kibana for cluster level logging.

Is this possible to do this on the instance of Kubernetes that's shipped with Docker for Windows as per the documentation? I'm not interested in third party Kubernetes manifests or Helm charts that mimic this behavior.

-- Bart
docker
kubernetes

1 Answer

8/17/2018

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.

It is a complex environment with a huge amount of information regarding the state of cluster and events processed during execution of pods lifecycle and health checking off all nodes and whole Kubernetes cluster.

I do not have practice with Docker for Windows, so my point of view is based on Kubernetes with Linux containers perspective.

To collect and analyze all of this information there are some tools like Fluentd, Logstash and they are accompanied by tools such as Elasticsearch and Kibana. Those cluster-level log aggregation can be realized using Kubernetes orchestration framework. So we can expect that some running containers take care of gathering data and other containers take care of other aspects of abstractions like analyzing and presentation layer. Please notice that some solutions depend on cloud platform features where Kubernetes environment is running. For example, GCP offers Stackdriver Logging.

We can mention some layers of log probes and analyses:

  • monitoring a pod is the most rudimentary form of viewing Kubernetes logs. You use the kubectl commands to fetch log data for each pod individually. These logs are stored in the pod and when the pod dies, the logs die with them.

  • monitoring a node. Collected log for each node are stored in a JSON file. This file can get really large. Node-level logs are more persistent than pod-level ones.

  • monitoring a cluster. Kubernetes doesn’t provide a default logging mechanism for the entire cluster, but leaves this up to the user and third-party tools to figure out. One approach is to build on the node-level logging. This way, you can assign an agent to log every node and combine their output.

As you see, there is a niche on cluster level monitoring, so there is a reason to aggregate current logs and offer a practical way to analyze and present results.

On the node level logging, popular log aggregator is Fluentd. It is implemented as a Docker container, and it is run parallel with pod lifecycle. Fluentd does not store the logs themselves. Instead, it sends their logs to an Elasticsearch cluster that stores the log information in a replicated set of nodes. It looks like Elasticsearch is used as a data store of aggregated logs of working nodes. This aggregator cluster consists of a pod with two instances of Elasticsearch.

The aggregated logs in the Elasticsearch cluster can be viewed using Kibana. This presents a web interface, which provides a more convenient interactive method for querying the ingested logs The Kibana pods are also monitored by the Kubernetes system to ensure they are running healthily and the expected number of replicas are present. The lifecycle of these pods is controlled by a replication-controller specification similar in nature to how the Elasticsearch cluster was configured.

Back to your question. I'm pretty sure that the mentioned above also works with Kubernetes and Dockers for Windows. From the other hand, I think the cloud platform or the Linux premise environment is a natural space to live for them.

Answer was inspired by Cluster-level Logging of Containers with Containers and Kubernetes Logging articles.

I also like Configuring centralized logging from Kubernetes page and used An Introduction to logging in Kubernetes at my beginning with Kubernetes.

-- d0bry
Source: StackOverflow