We re trying to eliminate Datadog agents from our infrastructure. I am trying to find a solution to forward the containers standard output logs to be visualised on datadog but without the agents and without changing the dockerfiles because there are hundreds of them.
I was thinking about trying to centralize the logs with rsyslog but I dont know if its a good idea. Any suggestions ?
This doc will show you a comprehensive list of all integrations that involve log collection. Some of these include other common log shippers, which can also be used to forward logs to Datadog. Among these you'd find...
That said, you can still just use the Datadog agent to collect logs only (they want you to collect everything with their agent, that's why they warn you against collecting just their logs).
If you want to collect logs from docker containers, the Datadog agent is an easy way to do that, and it has the benefit of adding lots of relevant docker-metadata as tags to your logs. (Docker log collection instructions here.)
If you don't want to do that, I'd look at Fluentd first on the list above -- it has a good reputation for containerized log collection, promotes JSON log formatting (for easier processing), and scales reasonably well.