Helm timeout with EKS

5/29/2020

When we use helm to create Istio VirtualService or DestinationRules, it takes more than 30seconds. So we have set up a higher timeout of 5m.

Command used helm upgrade --install --wait --timeout 5m --v 9 helloworld ./templateDir

However, API Server call times out at 30s instead of waiting for 5m. I have pasted the log from helm below. When we use kubectl to create the same, we set --request-timeout param for kubectl and everything is fine

Since it is a timeout from apiserver, is it possible to set a higher timeout value for all API server requests from helm? Is there any other workaround you can suggest for this problem ?

I0528 17:52:57.664992   11148 round_trippers.go:423] curl -k -v -XPOST  -H "Content-Type: application/json" -H "Accept: application/json" 'https://aaaa.sk1.us-east-1.eks.amazonaws.com/apis/networking.istio.io/v1beta1/namespaces/default/destinationrules '
I0528 17:53:27.981691   11148 round_trippers.go:443] POST https://aaaa.sk1.us-east-1.eks.amazonaws.com/apis/networking.istio.io/v1beta1/namespaces/default/destinationrules  504 Gateway Timeout in 30316 milliseconds
I0528 17:53:27.981691   11148 round_trippers.go:449] Response Headers:
I0528 17:53:27.981691   11148 round_trippers.go:452]     Audit-Id: cba23005-b8db-47f7-8801-4c89e6447cd3
I0528 17:53:27.981691   11148 round_trippers.go:452]     Content-Type: application/json
I0528 17:53:27.981691   11148 round_trippers.go:452]     Content-Length: 187
I0528 17:53:27.981691   11148 round_trippers.go:452]     Date: Thu, 28 May 2020 12:23:27 GMT
I0528 17:53:27.981691   11148 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Timeout: request did not complete within requested timeout 30s","reason":"Timeout","details":{},"code":504}
I0528 17:53:27.982759   11148 request.go:1017] Request Body: {"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"name":"mbrsvc","namespace":"default"},"spec":{"hosts":["mbrsvc"],"http":[{"route":[{"destination":{"host":"mbrsvc","subset":"1.0.0"},"weight":100}]}]}}
-- RDR
amazon-eks
aws-eks
eks
kubernetes
kubernetes-helm

1 Answer

5/29/2020

Documentation states that the --timeout should be in seconds.

--timeout: A value in seconds to wait for Kubernetes commands to complete This defaults to 5m0s

--wait: Waits until all Pods are in a ready state, PVCs are bound, Deployments have minimum (Desired minus maxUnavailable) Pods in ready state and Services have an IP address (and Ingress if a LoadBalancer) before marking the release as successful. It will wait for as long as the --timeout value. If timeout is reached, the release will be marked as FAILED. Note: In scenario where Deployment has replicas set to 1 and maxUnavailable is not set to 0 as part of rolling update strategy, --wait will return as ready as it has satisfied the minimum Pod in ready condition.

Hope that helps.

-- Crou
Source: StackOverflow