How to push fluent logs from one K8s Cluster to ElasticSearch in another K8s Cluster?
I have a scenario which we have fluent daemons running on multiple K8s clusters (C-1, C-2, C-3) and we have ElasticSearch running on another K8s Cluster (C-4). We wanted to push all the fluent logs from C-1, C-2 and C-3 to elastic search on C-4.
Here is what I did:
Created an ingress host entry with ES service in C-4 cluster which looks like
es-entrypoint.logging.svc.cluster.local
The above ingress host can be accessible from all the cluster using master IP of the C4 cluster.
$curl -H "Host: es-entrypoint.logging.svc.cluster.local" http://10.11.1.44
{
"name" : "elasticsearch-logging-p57wl",
"cluster_name" : "kubernetes-logging",
"cluster_uuid" : "ox8jE56mTg630zdKtThPHQ",
"version" : {
"number" : "6.3.2",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "053779d",
"build_date" : "2018-07-20T05:20:23.451332Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
---
apiVersion: v1
kind: Endpoints
metadata:
name: es-entrypoint
namespace: logging
subsets:
- addresses:
- ip: 10.11.1.44
ports:
- name: es-http
port: 80
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: es-entrypoint
namespace: logging
spec:
ports:
- name: es-http
port: 80
protocol: TCP
targetPort: 80
sessionAffinity: None
type: ClusterIP
Now I can access ingress Host uri from the pod in C-1, C-2 and C-3 clusters by using curl http://es-entrypoint.logging.svc.cluster.local:80
. This works well with kibana and all the pods, except flunetD.
Error
2019-06-13 06:06:57 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=10 next_retry_seconds=2019-06-13 06:07:31 +0000 chunk="58b2ddd4f511bc47e5bf7705f177cb2b" error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>\"es-entrypoint.logging.svc.cluster.local\", :port=>80, :scheme=>\"http\"})!"
Please find the fluent yml:
Any Idea ??