Http Read timeouts on query calls to Elastic Search cluster

11/27/2019

I'm troubleshooting an issue with our application hosted in K8s pod, which tries to query ES cluster through ES Jest Client. Most of the queries were successful, but randomly a call fails with read timeout. Any help will be appreciated.

Jest Client version:6.3.1

ES cluster is hosted in AWS

Below i'm providing apache http debug logs.

o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context 
o.a.h.i.c.BasicHttpClientConnectionManager - Get connection for route
{}->http://dev-app-elasticsearch-lb.xyz.com:9200  
o.a.h.impl.execchain.MainClientExec - Executing request POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1   o.a.h.impl.execchain.MainClientExec - Target auth state:
UNCHALLENGED   o.a.h.impl.execchain.MainClientExec - Proxy auth state:
UNCHALLENGED   org.apache.http.headers - http-outgoing-16 >> POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1   org.apache.http.headers - http-outgoing-16 >>
Content-Length: 2531   org.apache.http.headers - http-outgoing-16 >>
Content-Type: application/json; charset=UTF-8  
org.apache.http.headers - http-outgoing-16 >> Host:
dev-app-elasticsearch-lb.xyz.com:9200   org.apache.http.headers -
http-outgoing-16 >> Connection: Keep-Alive   org.apache.http.headers -
http-outgoing-16 >> User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181)
org.apache.http.headers - http-outgoing-16 >> Accept-Encoding:
gzip,deflate   org.apache.http.wire - http-outgoing-16 >> "POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1[\r][\n]"   org.apache.http.wire - http-outgoing-16 >>
"Content-Length: 2531[\r][\n]"   org.apache.http.wire -
http-outgoing-16 >> "Content-Type: application/json;
charset=UTF-8[\r][\n]"   org.apache.http.wire - http-outgoing-16 >>
"Host: dev-app-elasticsearch-lb.xyz.com:9200[\r][\n]"  
org.apache.http.wire - http-outgoing-16 >> "Connection:
Keep-Alive[\r][\n]"   org.apache.http.wire - http-outgoing-16 >>
"User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181)[\r][\n]"  
org.apache.http.wire - http-outgoing-16 >> "Accept-Encoding:
gzip,deflate[\r][\n]"   org.apache.http.wire - http-outgoing-16 >>
"[\r][\n]"  org.apache.http.wire - http-outgoing-16 >> "{[\n]"
....<<outgoing query details>>  
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS  
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS  
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS  
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS  
c.t.c.util.concurrent.DatabaseLock - Cleaning up expired locks from DS
0   o.a.h.i.c.BasicHttpClientConnectionManager - Closing connection  
o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-6: Close
connection   o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing
connections idle longer than 15 MINUTES 
o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections
idle longer than 15 MINUTES  
o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections
idle longer than 15 MINUTES```

Any help would be greatly appreciated.

Thanks Naresh

-- naresh goty
elasticsearch
httpclient
jest
kubernetes

0 Answers