I'm troubleshooting an issue with our application hosted in K8s pod, which tries to query ES cluster through ES Jest Client. Most of the queries were successful, but randomly a call fails with read timeout. Any help will be appreciated.
Jest Client version:6.3.1
ES cluster is hosted in AWS
Below i'm providing apache http debug logs.
o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
o.a.h.i.c.BasicHttpClientConnectionManager - Get connection for route
{}->http://dev-app-elasticsearch-lb.xyz.com:9200
o.a.h.impl.execchain.MainClientExec - Executing request POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1 o.a.h.impl.execchain.MainClientExec - Target auth state:
UNCHALLENGED o.a.h.impl.execchain.MainClientExec - Proxy auth state:
UNCHALLENGED org.apache.http.headers - http-outgoing-16 >> POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1 org.apache.http.headers - http-outgoing-16 >>
Content-Length: 2531 org.apache.http.headers - http-outgoing-16 >>
Content-Type: application/json; charset=UTF-8
org.apache.http.headers - http-outgoing-16 >> Host:
dev-app-elasticsearch-lb.xyz.com:9200 org.apache.http.headers -
http-outgoing-16 >> Connection: Keep-Alive org.apache.http.headers -
http-outgoing-16 >> User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181)
org.apache.http.headers - http-outgoing-16 >> Accept-Encoding:
gzip,deflate org.apache.http.wire - http-outgoing-16 >> "POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"Content-Length: 2531[\r][\n]" org.apache.http.wire -
http-outgoing-16 >> "Content-Type: application/json;
charset=UTF-8[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"Host: dev-app-elasticsearch-lb.xyz.com:9200[\r][\n]"
org.apache.http.wire - http-outgoing-16 >> "Connection:
Keep-Alive[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181)[\r][\n]"
org.apache.http.wire - http-outgoing-16 >> "Accept-Encoding:
gzip,deflate[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "{[\n]"
....<<outgoing query details>>
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
c.t.c.util.concurrent.DatabaseLock - Cleaning up expired locks from DS
0 o.a.h.i.c.BasicHttpClientConnectionManager - Closing connection
o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-6: Close
connection o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing
connections idle longer than 15 MINUTES
o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections
idle longer than 15 MINUTES
o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections
idle longer than 15 MINUTES```
Any help would be greatly appreciated.
Thanks Naresh