Fluentbit Hangs after some time in EFK stack

5/31/2019

I'm running a Kubernetes Cluster with EFK stack and after some time the logs seems to go missing in Kibana.

When I check the logs manually of a fluent bit pod, the following result was shown.

However the logs start re appearing when I restart the fluent-bit pods. And the space of my volume is not full either. Any ideas on how to resolve this?

[2019/05/31 02:28:13] [error] [out_es] could not pack/validate JSON response
{"took":0,"errors":true,"items":[{"index":{"_index":"kubernetes_cluster-2019.05.27","_type":"flb_type","_id":"m5a3C2sBq0LHM0frVg1S","status":403,"error":{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}}},{"index":{"_index":"kubernetes_cluster-2019.05.27","_type":"flb_type","_id":"nJa3C2sBq0LHM0frVg1S","status":403,"error":{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}}},{"index":{"_index":"kubernetes_cluster-2019.05.27","_type":"flb_type","_id":"nZa3C2sBq0LHM0frVg1S","status":403,"error":{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}}},{"index":{"_index":"kubernetes_cluster-2019.05.27","_type":"flb_type","_id":"npa3C2sBq0LHM0frVg1S","status":403,"error":{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-onl
[2019/05/31 02:28:13] [ warn] [out_es] Elasticsearch error``` .
-- nilesh93
efk
elasticsearch
fluent-bit
kubernetes

0 Answers