Unable to create the fluentd containers in my kubernetes cluster on ubuntu

10/10/2018

I am trying to do the log monitoring of my kubernetes cluster using Elasticsearch, Fluentd, and Kibana. Here is the link which I was followed in this task. I labeled the nodes with beta.kubernetes.io/fluentd-ds-ready: "true". Initially, I created the statefulset for Elasticsearch.

After that, I created the fluentd-es-configmap.yaml,fluentd-es-ds.yaml and checked the pods status using kubectl get pods -n kube-system. The Fluentd pods are showing status like container creating. I checked the logs of the Fluentd container and it shows the error like:

Error from server (BadRequest): container "fluentd-es" in pod "fluentd-es-v2.0.1-csx96" is waiting to start: ContainerCreating

Here is fluentd pod description:

Name:               fluentd-es-v2.0.1-csx96
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               ldap/192.168.1.191
Start Time:         Wed, 10 Oct 2018 15:08:17 -0400
Labels:             controller-revision-hash=5754d85c97
                    k8s-app=fluentd-es
                    kubernetes.io/cluster-service=true
                    pod-template-generation=1
                    version=v2.0.1
Annotations:        scheduler.alpha.kubernetes.io/critical-pod:
Status:             Pending
IP:
Controlled By:      DaemonSet/fluentd-es-v2.0.1
Containers:
  fluentd-es:
    Container ID:
    Image:          gcr.io/google-containers/fluentd-elasticsearch:v2.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  200Mi
    Environment:
      FLUENTD_ARGS:  --no-supervisor -q
    Mounts:
      /etc/fluent/config.d from config-volume (rw)
      /host/lib from libsystemddir (ro)
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from fluentd-es-token-l2b2m                                                                                         (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:
  varlibdockercontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:
  libsystemddir:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/lib64
    HostPathType:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      fluentd-es-config-v0.1.0
    Optional:  false
  fluentd-es-token-l2b2m:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  fluentd-es-token-l2b2m
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/fluentd-ds-ready=true
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason       Age                    From           Message
  ----     ------       ----                   ----           -------
  Warning  FailedMount  14m (x42 over 107m)    kubelet, ldap  Unable to mount vo                                                                                        lumes for pod "fluentd-es-v2.0.1-csx96_kube-system(d80d9c78-ccbf-11e8-b7b5-52540                                                                                        0e4ff36)": timeout expired waiting for volumes to attach or mount for pod "kube-                                                                                        system"/"fluentd-es-v2.0.1-csx96". list of unmounted volumes=[config-volume]. li                                                                                        st of unattached volumes=[varlog varlibdockercontainers libsystemddir config-volume fluentd-es-token-l2b2m]                                                                                    
  Warning  FailedMount  3m23s (x60 over 109m)  kubelet, ldap  MountVolume.SetUp  failed for volume "config-volume" : configmap "fluentd-es-config-v0.1.0" not found                                                                                                                                                                             

Could anybody suggest me how to resolve this issue?

Thanks in advance.

-- gayathri
elasticsearch
fluentd
kubernetes

1 Answer

10/10/2018

The problem seems to be a mismatch in the name of the configmap. The DaemonSet in looking for a configmap named fluentd-es-config-v0.1.0 but it is not found.

In the repository the configmap is named fluentd-es-config-v0.1.5 in both fluentd-es-ds.yaml and fluentd-es-configmap.yaml, so it should work by just using these files.

-- Ignacio Millán
Source: StackOverflow