I have a java API (accepting HTTPS requests_ packaged into a docker image, and then it is deployed using k8s cluster on top of EC2s. The master EC2 has an ELB in front.
I can make curl POST requests to the ELB in order to hit that java API.
Sometimes my curl request sits waiting for a response forever even though when i see the kube logs the processing was successful.
This happens for larger requests around 40mins, requests of 25mins are getting a response ok.
Where do you think the timeout could be? any specific config params i should look at?
client (curl) --> ELB --> k8s --> pod running a java api image
i thought this would be relevant (i am not setting IdleTimeout) for ELB but docs say default is 60s, although i can get response for 20min requests "ConnectionSettings": { "IdleTimeout" }
The ELB timeout counts only for "idle" time. That means as long as your upload is still running, it's not idle. When the file arrived at your server, you need to measure the time until your server responds.
When got that correctly, the server will process the request and return a response to the client afterwards. With a default timeout of 60 seconds, your server has these 60 seconds to process the uploaded file and return an answer.
Maybe your server needs less than 60 seconds to process your 25 minutes upload, but more to process the 40 minutes upload?
aws elb modify-load-balancer-attributes --load-balancer-name my-loadbalancer --load-balancer-attributes "{\"ConnectionSettings\":{\"IdleTimeout\":300}}"use this in cLI to change timeout to 5 min
Why dont you upload the file and push an event to message broker like rabbitMQ. Perform ETL on the files using kubernetes Job/CronJob object asynchronously and accordingly notify the client.
This way, you dont have to block the incoming request for longer time. post upload after the event is published send a message to client that request is being processed.
25 minutes is quite long for an HTTP request. I quite agree with P Ekambaram.
I think it could be better to make the endpoint asynchronous and reply as soon as the file is uploaded (should be quicker), at the same time, use middleware messaging (RabbitMQ, Kafka or NATS) or Websocket, that push an event once the file is successfully imported and processed.
Just as Pampy mentioned in his answer, the ELB timeout counts only the Idle time. This can range between 1 and 4000 secs and is set to 60 secs by default. You can change the timeout using the CLI or the console.
The following is an example of using the CLI to change it to 5 mins:
aws elb modify-load-balancer-attributes --load-balancer-name my-loadbalancer --load-balancer-attributes "{\"ConnectionSettings\":{\"IdleTimeout\":300}}"Source: docs
As you are uploading big files taking 20-40 mins, I would still recommend the other suggestions about using a message broker like RabbitM or Kafka to handle the upload and processing asynchronously.