Azure AKS pod logs are shown with 30+ minute delay in Log Analytics

5/6/2020

There is an AKS cluster in Azure with a microservices app (NodeJS) deployed.
The app is streaming logs to Application Insights (and to Log Analytics) using application insights.
All settings are by default: maxBatchSize & maxBatchIntervalMs.

When I check the pod logs using kubectl logs {POD_NAME} -n {NAMESPACE_NAME} I see the following output:

---

2020-05-05T00:22:24.851Z info xxxxxxxxx-a94b-4666-9e83-31e945e1ee15 0oaXXXXXXXXXX PUT /storages/1XXXXXX007475 xxxMiddleware

xxxxMiddleware ended
Postgres pool raised an error. Error: read ETIMEDOUT
---

In Log Analytics using Kusto query I see the same:

LogEntrySource  stderr
LogEntry    Postgres pool raised an error. Error: read ETIMEDOUT
TimeGenerated   2020-05-05T00:58:09
Computer    aks-agentpool-xxxxx-3
ContainerID 4a5e5e69957917578d8b18b59628ec3f21cd1dd19b5d66ca1xxxxxxxxxx

I'm wondering why the difference in timestamps is ~ 35+ minutes, i.e. 2020-05-05T00:58:09 - 2020-05-05T00:22:24.851Z

-- Sergey
azure
azure-aks
azure-log-analytics
kubernetes

1 Answer

5/7/2020

Yes, the difference is about 30 min. I guess this could be the difference in the timezone. kubectl reports logs in the timezone of the pod where as LogAnalytics might be reports it into UTC timezone or in your browser timezone. Please check the timezones of pod and the browser and see if this makes sense.

Hopefully this should solve the problem.

-- Atul
Source: StackOverflow