ApplicationInsight not getting telemetry from AKS pods

9/8/2021

In Application insight and dashboards, we are getting telemetry from the API manager APIS. But in the same App insight we are not getting telemtry from the backend applications which is running in AKS pods and connected using instarumentation key as environment variables inside the containers. in the application pod logs we are getting the error as given below and its noticed that new releases of applications are only affected by this issue and some applications which was not recently deployed not faing issue. We checked the instrumentation key and string part in the affected pods contaners and all have proper environment variable set for the instrumentation key. Not sure about the root cause here whether there are any other factors will come in picture for the below issue otherthan the instrumentation key to configure the connection between aks and Application insight?

some of the pods we are getting below error

WARN c.m.a.i.c.c.TransmissionNetworkOutput - In the last 5 minutes, the following operation has failed 1 times (out of 42): Unable to send telemetry to the ingestion service (telemetry will be stored to disk):

  • I/O exception: org.apache.http.NoHttpResponseException: dc.services.visualstudio.com:443 failed to respond (1 times) WARN c.m.a.i.c.c.TransmissionNetworkOutput - In the last 5 minutes, the following operation has failed 1 times (out of 41): Unable to send telemetry to the ingestion service (telemetry will be stored to disk):

however when we tested the communication from the node where the pods were running to dc.services.visualstudio.com:443 by telnet, its getting connected.

other pods we are getting this error,

  • ingestion service returned 400 (Telemetry item length must not exceed 65536) (1 times) WARN c.m.a.i.c.c.TransmissionNetworkOutput - In the last 5 minutes, the following operation has failed 1 times (out of 1): Unable to send telemetry to the ingestion service (telemetry will be stored to disk): * ingestion service returned 400 (Telemetry item length must not exceed 65536) (1 times) WARN c.m.a.i.c.c.TransmissionNetworkOutput - In the last 5 minutes, the following operation has failed 1 times (out of 1): Unable to send telemetry to the ingestion service (telemetry will be stored to disk):
-- Vowneee
appinsights
azure-aks
azure-application-insights
azure-application-insights-profiler
kubernetes

1 Answer

9/9/2021

Have you installed the package Application Insights for Kubernetes

TelemetryEnhancement.png

For Installing Packages Check Microsoft Application Insights for Kubernetes

Also Check the similar SO for the required information about Log Quarries.

-- SaiSakethGuduru-MT
Source: StackOverflow