ElasticsearchJS timing out on Kubernetes Google Cloud?

5/6/2018

I have a NodeJS application running on a pod in GKE and for some reason my application is timing out on all requests to Elasticsearch (which is a VM in the same zone as the Kubernetes cluster). What's weird is that if I create a simple client with the same settings used by my application (directly in the container, via node repl), all requests will be completed.

I noticed that my application was able to request Elasticsearch and receive a response at startup (when there is no HTTP connection to the application). It is a simple REST endpoint. Here is the container log:

[2018-05-06T07:46:14.906Z] TRACE (identity/26 on identity-deployment-54d5c7ccb9-hj2k6): REST request
req: {
  "headers": {
    "host": "research.evosec.ws",
    "connection": "close",
    "x-real-ip": "201.6.141.44",
    "x-forwarded-for": "201.6.141.44",
    "x-forwarded-host": "research.evosec.ws",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-original-uri": "/identity/user",
    "x-scheme": "https",
    "x-original-forwarded-for": "201.6.141.44, 35.201.82.80",
    "content-type": "application/json",
    "cache-control": "no-cache",
    "postman-token": "5646d274-4ed3-44d8-952a-abab50ff7f52",
    "user-agent": "PostmanRuntime/7.1.1",
    "accept": "*/*",
    "accept-encoding": "gzip, deflate",
    "x-cloud-trace-context": "52330745e4ee03559f557b56026f1808/1724170532745914334",
    "via": "1.1 google"
  },
  "method": "GET",
  "remoteAddress": "201.6.141.44",
  "url": "/user"
}
[2018-05-06T07:46:14.908Z] DEBUG (elasticsearch/26 on identity-deployment-54d5c7ccb9-hj2k6): starting request {"method":"POST","path":"/identity_user/user/_search","body":{"query":{"bool":{"filter":{"bool":{"must":[{"term":{"trash":false}},{"term":{"document_type.keyword":"user"}}]}}}}},"query":{"from":0,"size":10,"_source_include":"nickname,username,email,is_email_validated,email_validation_date,system_ownership,user_groups,roles,trash,created_at,last_update"}}
[2018-05-06T07:46:44.910Z] TRACE (identity/26 on identity-deployment-54d5c7ccb9-hj2k6): Return response error
error: {
  "message": "Request Timeout after 30000ms",
  "stack":
      Error: Request Timeout after 30000ms
          at /usr/src/app/node_modules/elasticsearch/src/lib/transport.js:354:15
          at Timeout.<anonymous> (/usr/src/app/node_modules/elasticsearch/src/lib/transport.js:383:7)
          at ontimeout (timers.js:482:11)
          at tryOnTimeout (timers.js:317:5)
          at Timer.listOnTimeout (timers.js:277:5)
}
[2018-05-06T07:46:44.911Z] ERROR (identity/26 on identity-deployment-54d5c7ccb9-hj2k6): Server error
req: {
  "headers": {
    "host": "research.evosec.ws",
    "connection": "close",
    "x-real-ip": "201.6.141.44",
    "x-forwarded-for": "201.6.141.44",
    "x-forwarded-host": "research.evosec.ws",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-original-uri": "/identity/user",
    "x-scheme": "https",
    "x-original-forwarded-for": "201.6.141.44, 35.201.82.80",
    "content-type": "application/json",
    "cache-control": "no-cache",
    "postman-token": "5646d274-4ed3-44d8-952a-abab50ff7f52",
    "user-agent": "PostmanRuntime/7.1.1",
    "accept": "*/*",
    "accept-encoding": "gzip, deflate",
    "x-cloud-trace-context": "52330745e4ee03559f557b56026f1808/1724170532745914334",
    "via": "1.1 google"
  },
  "method": "GET",
  "remoteAddress": "201.6.141.44",
  "url": "/user"
}
resHeaders: {
  "content-type": "application/json",
  "content-length": 77
}
resBody: {
  "errors": [
    {
      "displayName": "RequestTimeout",
      "message": "Request Timeout after 30000ms",
      "stack": "Error: Request Timeout after 30000ms\n    at /usr/src/app/node_modules/elasticsearch/src/lib/transport.js:354:15\n    at Timeout.<anonymous> (/usr/src/app/node_modules/elasticsearch/src/lib/transport.js:383:7)\n    at ontimeout (timers.js:482:11)\n    at tryOnTimeout (timers.js:317:5)\n    at Timer.listOnTimeout (timers.js:277:5)"
    }
  ]
}
[2018-05-06T07:46:44.912Z] TRACE (elasticsearch/26 on identity-deployment-54d5c7ccb9-hj2k6): 
method: "POST"
requestUrl: {
  "method": "POST",
  "protocol": "http:",
  "hostname": "elasticsearch-0",
  "port": 9200,
  "path": "/identity_user/user/_search?from=0&size=10&_source_include=nickname%2Cusername%2Cemail%2Cis_email_validated%2Cemail_validation_date%2Csystem_ownership%2Cuser_groups%2Croles%2Ctrash%2Ccreated_at%2Clast_update",
  "headers": {
    "content-type": "application/json"
  }
}
body: "{\"query\":{\"bool\":{\"filter\":{\"bool\":{\"must\":[{\"term\":{\"trash\":false}},{\"term\":{\"document_type.keyword\":\"user\"}}]}}}}}"
responseStatus: 0

Anyone have idea of what is happening?

EDIT

More information:

I monitored the connections of my application during a request using netstat. The connections to Kafka are ok, the request connection (port 8090) is ok, but the connection with Elasticsearch does not even appear!

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         
State       PID/Program name
tcp        0      0 10.0.2.14:60250         10.150.0.8:9092         
ESTABLISHED 25/node
tcp        0      0 10.0.2.14:44686         10.150.0.8:9092         
ESTABLISHED 26/node
tcp        0      0 10.0.2.14:60260         10.150.0.8:9092         
ESTABLISHED 25/node
tcp        0      0 10.0.2.14:8090          10.0.1.12:53234         
ESTABLISHED 26/node
tcp        0      0 10.0.2.14:60256         10.150.0.8:9092         
ESTABLISHED 26/node
-- Victor França
elasticsearch
google-kubernetes-engine
node.js

1 Answer

5/15/2018

You seem to be connecting through the external IP? If so, either

  1. Check if port 9200 is open in your firewall rules.
  2. Connect to your app through the internal IP (I understand they are in the same network).
-- suren
Source: StackOverflow