K
Q

Error extracting container id - source value does not contain matcher's logs_path '/var/lib/docker/containers/'

6/10/2020

I am collect container's log using filebeat in kubernetes cluster, and now collected log shows this error:

2020-06-10T09:26:35.831Z ERROR [kubernetes] add_kubernetes_metadata/matchers.go:91 Error extracting container id - source value does not contain matcher's logs_path '/var/lib/docker/containers/'.

this is the full log output:

enter image description here

I find the filebeat was listening is the node meowk8sslave2 and login into this node found the path exists. why the error could happen? this is my filebeat config:

{
	"filebeat.yml": "filebeat.inputs:
		- type: container
		  paths:
		    - /var/log/containers/*.log
		  processors:
		  - add_kubernetes_metadata:
		      host: ${NODE_NAME}
		      matchers:
		      - logs_path:
		          logs_path: \"/var/log/containers/\"
		
		output.elasticsearch:
		  host: '${NODE_NAME}'
		  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
		"
}
-- Dolphin
kubernetes

Similar Questions

kubectl - cert manager - credentials not found
change the annotation for node-selector for openshift namespace
Setting up HTTP+HTTPs and TCP ports on a LoadBalancer (Want to forward GitLab SSH and Web)
How to find out why mounting an emptyDir volume fails in Kubernetes?
How to change Kubernetes configuration key file from controller-manager and apiserver to point to the same x509 RSA key?

3 Answers

6/23/2020

Look inside your filebeat pod where exactly the logs are made available.

I was testing the ELK stack on Minikube.

In my case it was inside /var/lib/docker/containers/*/*.log

So this one worked for me.

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/lib/docker/containers/*/*.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/lib/docker/containers/"
    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
-- Ashfaq
Source: StackOverflow
6/10/2020

change 

filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    # filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

to 

# filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/log/containers/*.log
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: container
           paths:
             - /var/log/containers/*${data.kubernetes.container.id}.log

Reference: https://discuss.elastic.co/t/problem-to-update-to-filebeat-7-7-0-and-parser-nginx-ingress-controller-on-kubernetes/232461/2

-- TeYoU
Source: StackOverflow
9/10/2021

works for me: \ update processors \ from:

processors:
  - add_cloud_metadata: ~
  - add_kubernetes_metadata:
        in_cluster: true
  - drop_event.when.regexp.message: "kube-probe"

to:

processors:
  - add_cloud_metadata: ~
  - add_kubernetes_metadata:
        in_cluster: true
        host: ${NODE_NAME}
        matchers:
        - logs_path:
              logs_path: "/var/log/containers/"
  - drop_event.when.regexp.message: "kube-probe"

maybe you need to update your nginx module log path to: \ /var/log/containers/*-${data.kubernetes.container.id}.log

-- exud
Source: StackOverflow