i am running an EFK-Stack (elastic, fluent-bit, kibana) on an azure kubernetes service. For some reasen i dont get the index lifecyle management to work properly. I added the Logstash_Format On to the output-elasticsearch.conf in order the create a new index every day, like logstash-* This is what my output-elasticsearch.conf looks like: `

    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWD}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False

In order to get ilm to work i followed the documentation provided by elastic (https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html). I added the following configuration according to the documentation: 1. add index lifecycle policy:

PUT _ilm/policy/logstash_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "5mb",
            "max_age": "1h"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

2. Create an index template

PUT _index_template/desc_template
{
  "index_patterns": ["logstash-*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logstash_policy",
      "index.lifecycle.rollover_alias": "logstash-delete"    
    }
  }
}

3. Bootstrap the initial index logstash with a write index alias

PUT logstash-000001
{
  "aliases": {
    "logstash": {
      "is_write_index": true
    }
  }
}

After reaching the 5mb border size, i set in the index policy, the new index logstash-000001 gets created properly. The only problem is, that the new rollover index remains empty. All the shiped logs from fluentbit gets still written to the daily index logstash-*. Am i missing something here. I also dont see any ilm config for fluent-bit available. Any help would very appreciated. Cheers Martin