I am trying to do the log monitoring of my kubernetes cluster using Elasticsearch, Fluentd, and Kibana. Here is the link which I was followed in this task. I labeled the nodes with beta.kubernetes.io/fluentd-ds-ready: "true". Initially, I created the statefulset for Elasticsearch.
After that, I created the fluentd-es-configmap.yaml,fluentd-es-ds.yaml and checked the pods status using kubectl get pods -n kube-system
. The Fluentd pods are showing status like container creating. I checked the logs of the Fluentd container and it shows the error like:
Error from server (BadRequest): container "fluentd-es" in pod "fluentd-es-v2.0.1-csx96" is waiting to start: ContainerCreating
Here is fluentd pod description:
Name: fluentd-es-v2.0.1-csx96
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: ldap/192.168.1.191
Start Time: Wed, 10 Oct 2018 15:08:17 -0400
Labels: controller-revision-hash=5754d85c97
k8s-app=fluentd-es
kubernetes.io/cluster-service=true
pod-template-generation=1
version=v2.0.1
Annotations: scheduler.alpha.kubernetes.io/critical-pod:
Status: Pending
IP:
Controlled By: DaemonSet/fluentd-es-v2.0.1
Containers:
fluentd-es:
Container ID:
Image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
memory: 500Mi
Requests:
cpu: 100m
memory: 200Mi
Environment:
FLUENTD_ARGS: --no-supervisor -q
Mounts:
/etc/fluent/config.d from config-volume (rw)
/host/lib from libsystemddir (ro)
/var/lib/docker/containers from varlibdockercontainers (ro)
/var/log from varlog (rw)
/var/run/secrets/kubernetes.io/serviceaccount from fluentd-es-token-l2b2m (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
varlibdockercontainers:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
HostPathType:
libsystemddir:
Type: HostPath (bare host directory volume)
Path: /usr/lib64
HostPathType:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: fluentd-es-config-v0.1.0
Optional: false
fluentd-es-token-l2b2m:
Type: Secret (a volume populated by a Secret)
SecretName: fluentd-es-token-l2b2m
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/fluentd-ds-ready=true
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 14m (x42 over 107m) kubelet, ldap Unable to mount vo lumes for pod "fluentd-es-v2.0.1-csx96_kube-system(d80d9c78-ccbf-11e8-b7b5-52540 0e4ff36)": timeout expired waiting for volumes to attach or mount for pod "kube- system"/"fluentd-es-v2.0.1-csx96". list of unmounted volumes=[config-volume]. li st of unattached volumes=[varlog varlibdockercontainers libsystemddir config-volume fluentd-es-token-l2b2m]
Warning FailedMount 3m23s (x60 over 109m) kubelet, ldap MountVolume.SetUp failed for volume "config-volume" : configmap "fluentd-es-config-v0.1.0" not found
Could anybody suggest me how to resolve this issue?
Thanks in advance.
The problem seems to be a mismatch in the name of the configmap. The DaemonSet in looking for a configmap named fluentd-es-config-v0.1.0
but it is not found.
In the repository the configmap is named fluentd-es-config-v0.1.5
in both fluentd-es-ds.yaml and fluentd-es-configmap.yaml, so it should work by just using these files.