K
Q

Question

Kubernetes on AWS with kube-up.sh no logs

9/4/2016

I'm setting up a cluster using kube-up.sh (v1.3.4) with AWS as provider. Everything is good, my only issue is that I don't see my logs (in Kibana and Elsaticsearch). If I docker attach inside the fluentd-elasticsearch container here is what I have:

in /var/logs/containers I see a symlink to a log file for every pods running on my node, for example:

elasticsearch-logging-v1-9lwly_kube-system_POD-3bb54515d4e0e479c39cb7c35aa6675fdfe2970233e095420e2116c2f633ab50.log

is a symlink to

/mnt/ephemeral/docker/containers/3bb54515d4e0e479c39cb7c35aa6675fdfe2970233e095420e2116c2f633ab50/3bb54515d4e0e479c39cb7c35aa6675fdfe2970233e095420e2116c2f633ab50-json.log

However, /mnt is empty in my fluentd-elasticsearch container. So all symlinks are wrong.

I guess it's just that /mnt/ephemeral is not bind mounted in the fluentd-elasticsearch container. I'm not advanced enough in kubernetes to know where to fix that, and might be a bug since I used the standard procedure to setup my cluster.

If I run kubectl describe on the pod I get:

Name:       fluentd-elasticsearch-ip-172-20-0-168.eu-west-1.compute.internal
Namespace:  kube-system
Node:       ip-172-20-0-168.eu-west-1.compute.internal/172.20.0.168
Start Time: Fri, 02 Sep 2016 17:15:03 +0200
Labels:     k8s-app=fluentd-logging
Status:     Running
IP:     10.244.3.2
Controllers:    <none>
Containers:
  fluentd-elasticsearch:
    Container ID:   docker://d376a900ef770e65dfbf75bf5bc1c711c650868bb4e4ea74002818852a81aa04
    Image:      gcr.io/google_containers/fluentd-elasticsearch:1.17
    Image ID:       docker://sha256:e74f564e4c316e0f6baebf838015f516e26d7501c96ead5f115523ff80c614fd
    Port:       
    Limits:
      memory:   200Mi
    Requests:
      cpu:          100m
      memory:           200Mi
    State:          Running
      Started:          Fri, 02 Sep 2016 17:15:36 +0200
    Ready:          True
    Restart Count:      0
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True 
  Ready     True 
  PodScheduled  True 
Volumes:
  varlog:
    Type:   HostPath (bare host directory volume)
    Path:   /var/log
  varlibdockercontainers:
    Type:   HostPath (bare host directory volume)
    Path:   /var/lib/docker/containers
QoS Tier:   Burstable
No events.

So obviously volumes are wrong (/var/lib/docker/containers instead of /mnt/ephemeral/docker/containers.

Where / how can I fix this ? Should this be reported as a bug in k8s repos ?

-- rmonjo

amazon-web-services

fluentd

kubernetes

logging

1 Answer

9/10/2016

I am using the cluster addon for a rackspace setup, not a aws one. But basiaclly you need to look for the kubernetes deployment file. In the rackspace case you need a daemon set, not a replication controller, I bet the AWS stuff is constructed the same way.

You need to add to the deployment yaml file:

containers:
        ...
        volumeMounts:
        - name: mntephemeraldockercontainers
          mountPath: /mnt/ephemeral/docker/containers
          readOnly: true
        ...
        volumes:
        - name: mntephemeraldockercontainers
          hostPath:
              path: /mnt/ephemeral/docker/containers

Why? Fluentd inside the pod read the path /var/log/containers/* which is a symlink that points to a "deak end" inside the container, because /mnt/ephemeral/docker/containers is not mounted there. It only exists on the host. So, mounting it in will make fluentd able to read the file.

NB: You need probably want to make the kubernetes metadata plugin work correctly, too.

rgds, j

-- Andreas John

Source: StackOverflow

KQ

Kubernetes on AWS with kube-up.sh no logs

Similar Questions

1 Answer

K
Q