K
Q

Logstash output s3 plugin is not working for kubernetes

6/19/2019

I am trying to send my kubernetes logs to s3 using logstash(v5.4.3) and logstash-output-s3 (5.0.7). logstash is creating logstash-programmatic-access-test-object but not the actual log files. Below are my input and output configurations.

I am trying to send my kubernetes logs to s3 using logstash(v5.4.3) and logstash-output-s3 (5.0.7). logstash is creating logstash-programmatic-access-test-object but not the actual log files. Below are my input and output configurations.

 input {
  file {
    # kubelet created symlinks to docker container logs
    path => "/var/log/containers/*.log"
    tags => ["kubernetes", "docker"]
    sincedb_path => "/var/log/containers/sincedb-kubernetes"
  }
}

output {
  if "kubernetes" and "docker" in [tags] {
     s3{
      access_key_id => "XXXXXXXXXX"
      secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXXX"
      region => "us-east-1"
      bucket => "nonapp-logs"
      size_file => 2048
      time_file => 5
      prefix => "non-app-logs-%{+YYYY.MM.dd}"
      canned_acl => "private"
      server_side_encryption => true
    }
  }
}

Below is the log when I am starting the service

[2019-06-19T11:49:52,648][DEBUG][logstash.agent           ] starting agent
[2019-06-19T11:49:52,649][DEBUG][logstash.agent           ] starting pipeline {:id=>"main"}
[2019-06-19T11:49:53,355][DEBUG][logstash.outputs.s3      ] Start periodic rotation check
[2019-06-19T11:49:53,364][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2019-06-19T11:49:53,775][INFO ][logstash.pipeline        ] Pipeline main started
[2019-06-19T11:49:53,806][DEBUG][logstash.agent           ] Starting puma
[2019-06-19T11:49:53,807][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2019-06-19T11:49:53,811][DEBUG][logstash.api.service     ] [api-service] start
[2019-06-19T11:49:53,818][DEBUG][logstash.inputs.file     ] _globbed_files: /var/log/containers/*.log: glob is: ["/var/log/containers/file1-42861859-dx660_default_file1-6faea70b0e43f4805378b02fcef43008c8a815b560bb6517b9e888a3622c24f1.log", "/var/log/containers/file2-n6dkl_monitoring_file2-26964aa335e8f60d7ed2e3fe1673346e4a33e322e1d5d42162b22a5b77f4f766.log", "/var/log/containers/aqua-agent-zck5m_file3-5d523fef15c1d4fc7979b11a33b39cb4e290b6a33b87c6f7789ed6be4849bd6a.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-98810bf8d063e535cd51f10be8b50a61a07922b5dd3ed8d20298f9af68dc56a2.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-48cd2f69df073460bbd0799ce9d918bfa4ac9587517e25b6cea9664939db4d59.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-5e7ff1e609f7cb858aeb17267d43c7f0602c6f79a9aef34ec7e0f0762f7bf4b1.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-98b868ffbc6d463092f634632040203ea3a8c349f667ff66d26562f52e4503b7.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-beeecf6ad217620a86f0ce5fb77820300df5a9a5019d0d5c1042987f37e6981d.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-021fef919f6892a5b343752851b96161b6f975a8457754f32a1ab9bf2d8d07da.log", "/var/log/containers/file2-n6dkl_monitoring_file2-1708ecd0708f31eaac2c0906a1cf0044152de4c750e2ba10fd56d32c7026779f.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-55209391560c6e544a4eacb54cc7d19588f7b263663a182da5092c5edb3911cf.log", "/var/log/containers/logstash-9k41j_kube-system_logstash-0577c19fe06f0de92a88b6f74a42313169945ca730f9f6c0c4aeb8de736bfccc.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-48a5402aa102d30d4b61c93130cb7501deb8e67620b17c68b65e713d9b6a4003.log", "/var/log/containers/file5-532757934-9vh43_default_file5-14d5f783f8f9836703006dff011ec68e8e84187a1027555c524c761087594a6f.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-demo-api-e3e65ab389eec9302aeb76ce34c0b08dc68088df7a3e19be7daee8e43fc32a93.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-ng-demo-2b678c4ef7bab59eb686fa6284d60477f5e26aa877c53e1efe72e2c297df87e9.log"]
[2019-06-19T11:49:53,901][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-19T11:49:58,808][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2019-06-19T11:50:03,811][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2019-06-19T11:50:07,823][DEBUG][logstash.inputs.file     ] _globbed_files: /var/log/containers/*.log: glob is: ["/var/log/containers/file1-42861859-dx660_default_file1-6faea70b0e43f4805378b02fcef43008c8a815b560bb6517b9e888a3622c24f1.log", "/var/log/containers/file2-n6dkl_monitoring_file2-26964aa335e8f60d7ed2e3fe1673346e4a33e322e1d5d42162b22a5b77f4f766.log", "/var/log/containers/aqua-agent-zck5m_file3-5d523fef15c1d4fc7979b11a33b39cb4e290b6a33b87c6f7789ed6be4849bd6a.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-98810bf8d063e535cd51f10be8b50a61a07922b5dd3ed8d20298f9af68dc56a2.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-48cd2f69df073460bbd0799ce9d918bfa4ac9587517e25b6cea9664939db4d59.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-5e7ff1e609f7cb858aeb17267d43c7f0602c6f79a9aef34ec7e0f0762f7bf4b1.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-98b868ffbc6d463092f634632040203ea3a8c349f667ff66d26562f52e4503b7.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-beeecf6ad217620a86f0ce5fb77820300df5a9a5019d0d5c1042987f37e6981d.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-021fef919f6892a5b343752851b96161b6f975a8457754f32a1ab9bf2d8d07da.log", "/var/log/containers/file2-n6dkl_monitoring_file2-1708ecd0708f31eaac2c0906a1cf0044152de4c750e2ba10fd56d32c7026779f.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-55209391560c6e544a4eacb54cc7d19588f7b263663a182da5092c5edb3911cf.log", "/var/log/containers/logstash-9k41j_kube-system_logstash-0577c19fe06f0de92a88b6f74a42313169945ca730f9f6c0c4aeb8de736bfccc.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-48a5402aa102d30d4b61c93130cb7501deb8e67620b17c68b65e713d9b6a4003.log", "/var/log/containers/file5-532757934-9vh43_default_file5-14d5f783f8f9836703006dff011ec68e8e84187a1027555c524c761087594a6f.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-demo-api-e3e65ab389eec9302aeb76ce34c0b08dc68088df7a3e19be7daee8e43fc32a93.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-ng-demo-2b678c4ef7bab59eb686fa6284d60477f5e26aa877c53e1efe72e2c297df87e9.log"]
[2019-06-19T11:50:08,362][DEBUG][logstash.outputs.s3      ] Periodic check for stale files
[2019-06-19T11:50:08,813][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2019-06-19T11:50:13,813][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2019-06-19T11:50:18,814][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2019-06-19T11:50:22,831][DEBUG][logstash.inputs.file     ] _globbed_files: /var/log/containers/*.log: glob is: ["/var/log/containers/file1-42861859-dx660_default_file1-6faea70b0e43f4805378b02fcef43008c8a815b560bb6517b9e888a3622c24f1.log", "/var/log/containers/file2-n6dkl_monitoring_file2-26964aa335e8f60d7ed2e3fe1673346e4a33e322e1d5d42162b22a5b77f4f766.log", "/var/log/containers/aqua-agent-zck5m_file3file3-5d523fef15c1d4fc7979b11a33b39cb4e290b6a33b87c6f7789ed6be4849bd6a.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-98810bf8d063e535cd51f10be8b50a61a07922b5dd3ed8d20298f9af68dc56a2.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-48cd2f69df073460bbd0799ce9d918bfa4ac9587517e25b6cea9664939db4d59.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-5e7ff1e609f7cb858aeb17267d43c7f0602c6f79a9aef34ec7e0f0762f7bf4b1.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-98b868ffbc6d463092f634632040203ea3a8c349f667ff66d26562f52e4503b7.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-beeecf6ad217620a86f0ce5fb77820300df5a9a5019d0d5c1042987f37e6981d.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-021fef919f6892a5b343752851b96161b6f975a8457754f32a1ab9bf2d8d07da.log", "/var/log/containers/file2-n6dkl_monitoring_file2-1708ecd0708f31eaac2c0906a1cf0044152de4c750e2ba10fd56d32c7026779f.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-55209391560c6e544a4eacb54cc7d19588f7b263663a182da5092c5edb3911cf.log", "/var/log/containers/logstash-9k41j_kube-system_logstash-0577c19fe06f0de92a88b6f74a42313169945ca730f9f6c0c4aeb8de736bfccc.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-48a5402aa102d30d4b61c93130cb7501deb8e67620b17c68b65e713d9b6a4003.log", "/var/log/containers/file5-532757934-9vh43_default_file5-14d5f783f8f9836703006dff011ec68e8e84187a1027555c524c761087594a6f.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-demo-api-e3e65ab389eec9302aeb76ce34c0b08dc68088df7a3e19be7daee8e43fc32a93.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-ng-demo-2b678c4ef7bab59eb686fa6284d60477f5e26aa877c53e1efe72e2c297df87e9.log"]
[2019-06-19T11:50:23,366][DEBUG][logstash.outputs.s3      ] Periodic check for stale files
[2019-06-19T11:50:23,814][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
^C[2019-06-19T11:50:23,887][WARN ][logstash.runner          ] SIGINT received. Shutting down the agent.
[2019-06-19T11:50:23,889][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2019-06-19T11:50:23,889][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2019-06-19T11:50:23,889][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2019-06-19T11:50:23,892][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2019-06-19T11:50:23,894][DEBUG][logstash.pipeline        ] Closing inputs
[2019-06-19T11:50:23,894][DEBUG][logstash.inputs.file     ] stopping {:plugin=>"LogStash::Inputs::File"}
[2019-06-19T11:50:23,894][DEBUG][logstash.pipeline        ] Closed inputs
[2019-06-19T11:50:24,850][DEBUG][logstash.inputs.file     ] closing {:plugin=>"LogStash::Inputs::File"}
[2019-06-19T11:50:24,851][DEBUG][logstash.pipeline        ] Input plugins stopped! Will shutdown filter/output workers.
[2019-06-19T11:50:24,918][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2019-06-19T11:50:24,919][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x624b35a1 run>"}
[2019-06-19T11:50:24,919][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x1e3416c8 sleep>"}
[2019-06-19T11:50:24,921][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x624b35a1>
[2019-06-19T11:50:24,955][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x1e3416c8>
[2019-06-19T11:50:24,956][DEBUG][logstash.outputs.s3      ] closing {:plugin=>"LogStash::Outputs::S3"}
[2019-06-19T11:50:24,956][DEBUG][logstash.outputs.s3      ] Uploading current workspace
[2019-06-19T11:50:24,961][DEBUG][logstash.pipeline        ] Pipeline main has been shutdown

    enter code here
-- Suveer Rao
amazon-s3
amazon-web-services
kubernetes
logstash

Similar Questions

Mounting kubernetes volume in multiple containers within a pod with gitlab
HPA not able to fetch metrics from Prometheus in Kubernetes
configure SSL in cluster of kubernetes
Handling CONNECT request with Nginx Ingress on GCP GKE

0 Answers