One of the pod replicas fails liveness probe and restarts

3/26/2021

Fluentd-elasticsearch pod is restarting too many times. Some of the pod don't have this problem. My guess is that Fluentd restarts because of some buffer problem.

When I look at the event log, it says

Container fluentd-elasticsearch failed liveness probe, will be restarted

Before the container restarts, it produces some logs, but I can't get what is wrong.

Logs: https://gist.github.com/eli-halych/a2ea69f708a83e07e907f79858941ba4#file-fluentd-elasticsearch-log

Another observation is that 2 replicas of the pod are running successfully. Only one replica has this problem and restarts many times.

Fluentd setup YAML: https://gist.github.com/eli-halych/a2ea69f708a83e07e907f79858941ba4#file-fluentd-setup-yaml


Helm chart used: https://github.com/kokuwaio/helm-charts/tree/main/charts/fluentd-elasticsearch

Fluentd version: 3.1.0

Elasticsearch version: 7.11.1

-- Eli Halych
elasticsearch
fluentd
kubernetes
kubernetes-helm
kubernetes-pod

0 Answers