How to push fluent logs from one K8s Cluster to ElasticSearch in another K8s Cluster?

6/13/2019

How to push fluent logs from one K8s Cluster to ElasticSearch in another K8s Cluster?

I have a scenario which we have fluent daemons running on multiple K8s clusters (C-1, C-2, C-3) and we have ElasticSearch running on another K8s Cluster (C-4). We wanted to push all the fluent logs from C-1, C-2 and C-3 to elastic search on C-4.

Here is what I did:

  1. Created an ingress host entry with ES service in C-4 cluster which looks like

    es-entrypoint.logging.svc.cluster.local

The above ingress host can be accessible from all the cluster using master IP of the C4 cluster.

$curl -H "Host: es-entrypoint.logging.svc.cluster.local" http://10.11.1.44

{
  "name" : "elasticsearch-logging-p57wl",
  "cluster_name" : "kubernetes-logging",
  "cluster_uuid" : "ox8jE56mTg630zdKtThPHQ",
  "version" : {
    "number" : "6.3.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "053779d",
    "build_date" : "2018-07-20T05:20:23.451332Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
  1. Then I have Created a Service without selector in C-1, C-2 and C-3 clusters.
---
apiVersion: v1
kind: Endpoints
metadata:
  name: es-entrypoint
  namespace: logging
subsets:
  - addresses:
    - ip: 10.11.1.44
    ports:
    - name: es-http
      port: 80
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: es-entrypoint
  namespace: logging
spec:
  ports:
  - name: es-http
    port: 80
    protocol: TCP
    targetPort: 80
  sessionAffinity: None
  type: ClusterIP

Now I can access ingress Host uri from the pod in C-1, C-2 and C-3 clusters by using curl http://es-entrypoint.logging.svc.cluster.local:80. This works well with kibana and all the pods, except flunetD.

Error

2019-06-13 06:06:57 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=10 next_retry_seconds=2019-06-13 06:07:31 +0000 chunk="58b2ddd4f511bc47e5bf7705f177cb2b" error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>\"es-entrypoint.logging.svc.cluster.local\", :port=>80, :scheme=>\"http\"})!"

Please find the fluent yml:

https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml

Any Idea ??

-- rolz
efk
elasticsearch
fluentd
kubernetes
kubernetes-ingress

0 Answers