Send logs from Kubernetes via Fluentd to Cloudwatch

6/4/2019

I am trying to understand cloud infra with Kubernetes and AWS

My trial here is to send logs to from my containers to cloud watch. Application (A sample Django with celery beat):

My pods:

NAMESPACE     NAME                                                                  READY   STATUS             RESTARTS   AGE
default       celery-beat-76dfdcc9d8-hrkzv                                          1/1     Running            0          93m
default       celery-worker-6f4d9f4985-66jfh                                        1/1     Running            0          93m
default       django-6db9754bd6-hnm84                                               1/1     Running            0          93m
default       django-migrations-9nb85                                               0/1     Completed          0          93m
default       flower-b977fcbd5-mdjrp                                                0/1     CrashLoopBackOff   22         93m
default       redis-fd65865fc-mfbwt                                                 1/1     Running            0          93m
kube-system   dns-controller-64b775df4d-trkvf                                       1/1     Running            1          9h
kube-system   etcd-manager-events-ip-172-20-45-193.us-west-1.compute.internal       1/1     Running            1          9h
kube-system   etcd-manager-main-ip-172-20-45-193.us-west-1.compute.internal         1/1     Running            1          9h
kube-system   kube-apiserver-ip-172-20-45-193.us-west-1.compute.internal            1/1     Running            3          9h
kube-system   kube-controller-manager-ip-172-20-45-193.us-west-1.compute.internal   1/1     Running            1          9h
kube-system   kube-dns-57dd96bb49-v6vqs                                             3/3     Running            3          9h
kube-system   kube-dns-57dd96bb49-vfrx9                                             3/3     Running            3          9h
kube-system   kube-dns-autoscaler-867b9fd49d-pg9j5                                  1/1     Running            1          9h
kube-system   kube-proxy-ip-172-20-45-193.us-west-1.compute.internal                1/1     Running            1          9h
kube-system   kube-proxy-ip-172-20-47-25.us-west-1.compute.internal                 1/1     Running            1          9h
kube-system   kube-proxy-ip-172-20-93-150.us-west-1.compute.internal                1/1     Running            1          9h
kube-system   kube-scheduler-ip-172-20-45-193.us-west-1.compute.internal            1/1     Running            1          9h
kube-system   kubernetes-dashboard-65c76f6c97-sdgjg                                 1/1     Running            1          9h
kube-system   tiller-deploy-58d57fd669-bzgfl                                        1/1     Running            1          9h

I able to access my application clusters with kubernetes dashboard. Under pods section - Logs was able to view couple of pods logs.

enter image description here

Now I am trying send these Logs to AWS CLoudwatch and then stream to AWS ELasticSearch

Following couple as per

Created a namespace logging

 **kubectl create namespace logging**
> helm init

my values.yaml

image:
  repository: fluent/fluentd-kubernetes-daemonset
  tag: v0.12.33-cloudwatch
## Specify an imagePullPolicy (Required)
## It's recommended to change this to 'Always' if the image tag is 'latest'
## ref: http://kubernetes.io/docs/user-guide/images/#updating-images
  pullPolicy: IfNotPresent

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  limits:
    cpu: 100m
    memory: 200Mi
  requests:
    cpu: 100m
    memory: 200Mi

# hostNetwork: false

## Add tolerations if specified
tolerations: []
#   - key: node-role.kubernetes.io/master
#     operator: Exists
#     effect: NoSchedule

annotations: {}

awsRegion: us-west-1
awsRole:
logGroupName: logging

Applied the changes (Running it as root)

helm install --name fluentd -f ./values.yaml incubator/fluentd-cloudwatch --namespace logging

When after execution the fluentd is reporting CrashLoopBackOff

logging fluentd-fluentd-cloudwatch-b9dlb
0/1 CrashLoopBackOff 1 16s logging
fluentd-fluentd-cloudwatch-jm229
0/1 CrashLoopBackOff 1 16s

Logs

kubectl logs fluentd-fluentd-cloudwatch-b9dlb --namespace logging

standard_init_linux.go:190: exec user process caused "no such file or directory"

Request some guidance

-- Ragav
amazon-cloudwatch
amazon-ec2
fluentd
kubernetes

0 Answers