K
Q

Can't harvest Nginx logs with Elastic Stack and Filebeat on ECK

7/24/2020

I'm new to Elastic Stack. I have to prepare a deployment of the Elastic stack with Filebeat under ECK (locally). To do this, I'm trying to retrieve logs from a Nginx server. I just do "F5 or Ctrl+F5" on the Welcome to nginx page to check if the data go to Kibana.

For the moment, I only get data from the kube-system namespace but no data from my namespace "beats".

Everything is running ready 1/1 and the volumes are OK too: I access them with Windows Explorer.

Here are the tools I use:

  • Win10 PRO
  • Docker Desktop
  • WSL1/Ubuntu 18.04
  • a namespace named beats with the following elements (see code snippets below):

Nginx:

---
apiVersion: v1
kind: Service
metadata:
  namespace: beats
  name: nginx
  labels:
    app: nginx-ns-beats
spec:
  type: LoadBalancer
  ports:
    - port: 80
      protocol: TCP
      targetPort: http
  selector:
    app: nginx-ns-beats

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: beats
  name: nginx-ns-beats
spec:
  selector:
    matchLabels:
      app: nginx-ns-beats
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-ns-beats
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - name: http
              containerPort: 80
          volumeMounts:
            - mountPath: "/var/log/nginx"
              name: nginx-data
      volumes:
        - name: nginx-data
          persistentVolumeClaim:
            claimName: nginx-data-pvc

Filebeat:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: beats
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          templates:
            - condition.contains:
                kubernetes.namespace: beats
              config:
                - module: nginx
                  access:
                    enabled: true
                    var.paths: ["/path/to/nginx-data-pv/access.log"]
                    subPath: access.log
                    tags: ["access"]
                  error:
                    enabled: true
                    var.paths: ["/path/to/nginx-data-pv/error.log"]
                    subPath: error.log
                    tags: ["error"]
    processors:
      - drop_event:
          when:
            or:
              - contains:
                  kubernetes.pod.name: "filebeat"
              - contains:
                  kubernetes.pod.name: "elasticsearch"
              - contains:
                  kubernetes.pod.name: "kibana"
              - contains:
                  kubernetes.pod.name: "logstash"
              - contains:
                  kubernetes.container.name: "dashboard"
              - contains:
                  kubernetes.container.name: "manager"
              
      - add_cloud_metadata:
      - add_host_metadata:

    output.logstash:
      hosts: ["logstash:5044"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: beats
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.8.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: elasticsearch-es-http
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: ELASTICSEARCH_USERNAME
              value: elastic
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: elastic
                  name: elasticsearch-es-elastic-user
            - name: NODE_NAME
              # value: elasticsearch-es-elasticsearch-0
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            #- name: es-certs
              #mountPath: /mnt/elastic/tls.crt
              #readOnly: true
              #subPath: tls.crt
      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate
        #- name: es-certs
          #secret:
            #secretName: elasticsearch-es-http-certs-public

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: beats
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""] # "" indicates the core API group
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: beats
  labels:
    k8s-app: filebeat

---

Logstash:

---
apiVersion: v1
kind: Service
metadata:
  namespace: beats
  labels:
    app: logstash
  name: logstash
spec:
  ports:
    - name: "25826"
      port: 25826
      targetPort: 25826
    - name: "5044"
      port: 5044
      targetPort: 5044
  selector:
    app: logstash
status:
  loadBalancer: {}

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: beats
  name: logstash-configmap
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    # all input will come from filebeat, no local logs

    input {
      beats {
        port => 5044
      }
    }

    filter {

    }

    output {
      if "nginx_test" in [tags] {
        elasticsearch {
          index => "nginx_test-%{[@metadata][beat]}-%{+YYYY.MM.dd-H.m}"
          hosts => [ "${ES_HOSTS}" ]
          user => "${ES_USER}"
          password => "${ES_PASSWORD}"
          cacert => '/etc/logstash/certificates/ca.crt'
        }
      }
    }

---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: logstash
  name: logstash
  namespace: beats
spec:
  containers:
    - image: docker.elastic.co/logstash/logstash:7.8.0
      name: logstash
      ports:
        - containerPort: 25826
        - containerPort: 5044
      env:
        - name: ES_HOSTS
          value: "https://elasticsearch-es-http:9200"
        - name: ES_USER
          value: "elastic"
        - name: ES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: elasticsearch-es-elastic-user
              key: elastic
      resources: {}
      volumeMounts:
        - name: config-volume
          mountPath: /usr/share/logstash/config
        - name: logstash-pipeline-volume
          mountPath: /usr/share/logstash/pipeline
        - name: cert-ca
          mountPath: "/etc/logstash/certificates"
          readOnly: true
  restartPolicy: OnFailure
  volumes:
    - name: config-volume
      configMap:
        name: logstash-configmap
        items:
          - key: logstash.yml
            path: logstash.yml
    - name: logstash-pipeline-volume
      configMap:
        name: logstash-configmap
        items:
          - key: logstash.conf
            path: logstash.conf
    - name: cert-ca
      secret:
        secretName: elasticsearch-es-http-certs-public

status: {}

Elasticsearch:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: beats
spec:
  version: 7.8.0

  nodeSets:
    - name: elasticsearch
      count: 1
      config:
        node.store.allow_mmap: false
        node.master: true
        node.data: true
        node.ingest: true
        xpack.security.authc:
          anonymous:
            username: anonymous
            roles: superuser
            authz_exception: false
      podTemplate:
        metadata:
          labels:
            app: elasticsearch
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 4Gi
                  cpu: 0.5
                limits:
                  memory: 4Gi
                  cpu: 1
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms2g -Xmx2g"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            storageClassName: es-data
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 5Gi

Kibana:

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: beats
spec:
  version: 7.8.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    service:
      spec:
        type: LoadBalancer

Volumes:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: es-data
  namespace: beats
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nginx-data
  namespace: beats
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---
# https://kubernetes.io/docs/concepts/storage/volumes/#local
apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-pv
  namespace: beats
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: es-data
  hostPath:
    path: /path/to/nginx-data-pv/

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-data-pv
  namespace: beats
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  #storageClassName: nginx-data
  storageClassName: ""
  hostPath:
    path: /path/to/nginx-data-pv/

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-data-pvc
  namespace: beats
spec:
  #storageClassName: nginx-data
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: nginx-data-pv

I used this command kubectl -n beats logs FILEBEAT_POD_ID and I got this:

2020-07-24T14:00:26.939Z	INFO	[publisher_pipeline_output]	pipeline/output.go:144	Connecting to backoff(async(tcp://logstash:5044))
2020-07-24T14:00:26.939Z	INFO	[publisher]	pipeline/retry.go:221	retryer: send unwait signal to consumer
2020-07-24T14:00:26.939Z	INFO	[publisher]	pipeline/retry.go:225	  done
2020-07-24T14:00:28.598Z	ERROR	[reader_json]	readjson/json.go:57	Error decoding JSON: invalid character 'W' looking for beginning of value
2020-07-24T14:00:28.616Z	ERROR	[reader_json]	readjson/json.go:57	Error decoding JSON: invalid character 'W' looking for beginning of value
2020-07-24T14:00:28.634Z	ERROR	[reader_json]	readjson/json.go:57	Error decoding JSON: invalid character 'W' looking for beginning of value
2020-07-24T14:00:28.941Z	INFO	log/input.go:152	Configured paths: [/PATH/TO/NGINX-DATA-PV/access.log]
[...]
2020-07-24T14:00:31.949Z	INFO	log/input.go:152	Configured paths: [/var/log/nginx/access.log*]
2020-07-24T14:00:31.950Z	INFO	log/input.go:152	Configured paths: [/PATH/TO/NGINX-DATA-PV/access.log]
2020-07-24T14:00:32.898Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:155	Failed to connect to backoff(async(tcp://logstash:5044)): dial tcp 10.103.207.209:5044: connect: connection refused
2020-07-24T14:00:32.898Z	INFO	[publisher_pipeline_output]	pipeline/output.go:146	Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 2 reconnect attempt(s)
2020-07-24T14:00:32.898Z	INFO	[publisher]	pipeline/retry.go:221	retryer: send unwait signal to consumer
2020-07-24T14:00:32.898Z	INFO	[publisher]	pipeline/retry.go:225	  done
[...]
2020-07-24T14:00:33.915Z	INFO	log/input.go:152	Configured paths: [/var/log/nginx/access.log*]
2020-07-24T14:00:33.917Z	INFO	log/input.go:152	Configured paths: [/PATH/TO/NGINX-DATA-PV/access.log]
2020-07-24T14:00:33.918Z	INFO	log/input.go:152	Configured paths: [/PATH/TO/NGINX-DATA-PV/error.log]
2020-07-24T14:00:33.918Z	INFO	log/input.go:152	Configured paths: [/var/log/nginx/access.log*]
2020-07-24T14:00:37.102Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:155	Failed to connect to backoff(async(tcp://logstash:5044)): dial tcp 10.103.207.209:5044: connect: connection refused
2020-07-24T14:00:37.102Z	INFO	[publisher_pipeline_output]	pipeline/output.go:146	Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 3 reconnect attempt(s)
2020-07-24T14:00:37.102Z	INFO	[publisher]	pipeline/retry.go:221	retryer: send unwait signal to consumer
2020-07-24T14:00:37.102Z	INFO	[publisher]	pipeline/retry.go:225	  done
2020-07-24T14:00:53.004Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:155	Failed to connect to backoff(async(tcp://logstash:5044)): dial tcp 10.103.207.209:5044: connect: connection refused
2020-07-24T14:00:53.004Z	INFO	[publisher_pipeline_output]	pipeline/output.go:146	Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 4 reconnect attempt(s)
2020-07-24T14:00:53.004Z	INFO	[publisher]	pipeline/retry.go:221	retryer: send unwait signal to consumer
2020-07-24T14:00:53.004Z	INFO	[publisher]	pipeline/retry.go:225	  done
2020-07-24T14:00:53.004Z	INFO	[publisher_pipeline_output]	pipeline/output.go:152	Connection to backoff(async(tcp://logstash:5044)) established

Feel free to ask me for more information.

Thank you in advance for any help you could give me.

Guillaume.

-- Lomig
elastic-stack
filebeat
kubernetes
nginx

Similar Questions

Will a connection to a K8 pod port be successful even if no container is mapped to the port?
Kubernetes pods getting auto-restarted on long GC cycles
Kubernetes: extract environment variables from a file when pod kicked off
Inconsistency in API data which is developed in Spring boot?
Kubelet Not Starting: Crashing with Exit Status: 255/n/a

0 Answers