I am using this guide to run filebeat on a Kubernetes cluster. https://www.elastic.co/guide/en/beats/filebeat/master/running-on-kubernetes.html#_kubernetes_deploy_manifests

filebeat version: 6.6.0

I updated config file with:

filebeat.yml: |-
filebeat.config:
  inputs:
    # Mounted `filebeat-inputs` configmap:
    path: ${path.config}/inputs.d/*.yml
    # Reload inputs configs as they change:
    reload.enabled: false
  modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false
# To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      hints.enabled: true

filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
- module: apache2
  access:
    enabled: true
    var.paths: ["/var/log/apache2/access.log*"]
  error:
    enabled: true
    var.paths: ["/var/log/apache2/error.log*"]

But, the logs from the PHP application (/var/log/apache2/error.log) are not being fetched by filebeat. I checked by execing into the filebeat pod and I see that apache2 and nginx modules are not enabled.

How can I set it up correctly in above yaml file.

UPDATE

I updated filebeat config file with below settings:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      templates:
        - condition:
          config:
            - type: docker
              containers.ids:
                - "${data.kubernetes.container.id}"
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
        - condition:
            equals:
              kubernetes.labels.app: "my-apache-app"
          config:
            - module: apache2
              log:
                input:
                  type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-modules
  namespace: default
  labels:
    k8s-app: filebeat
data:
  apache2.yml: |-
    - module: apache2
      access:
    enabled: true
      error:
    enabled: true
  nginx.yml: |-
    - module: nginx
      access:
    enabled: true

Now, I am logging apache errors in /dev/stderr so that I can see it thru kubectl logs. Logs are fetching over kibana dashboard. But, apache module is still noe visible.

I tried checking with ./filebeat modules list:

Enabled:
apache2
nginx

Disabled:

Kibana Dashboard