Kubernetes : deployment failed to rollout even after pod ready status is true

6/10/2019

I'm trying to deploy one of my application using circleCI workflows in following steps:

  • setup environment
  • build
  • Unit test
  • Integration test
  • Deploy

whenever my integration test takes around 7-8 mins, I'm getting timeout error on circleCI console.

deployment.extensions/demo-app created
Waiting for deployment "demo-app" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "demo-app" rollout to finish: 0 of 1 updated replicas are available...
Too long with no output (exceeded 10m0s)

Seems like there is some problem in rollout status command or it is not able to detect that my test has been finished and pod is in ready state.

I've checked the output of kubectl describe pod command as well and it is showing me the pod status is ready.

I've already tried increasing no_output_timeout in circleCI config but didn't work.

I've also tried changing readiness probe parameters' value according to time taken by integration test but still it finished with timeout. Current values for readiness probe:

readinessProbe:
    exec:
        command:
            - cat
            - /test/result
    initialDelaySeconds: 10
    periodSeconds: 10
    failureThreshold: 60

When I re-run the circleCI build after reducing number of integration tests, it is working fine.

Please let me know the probable cause for this as I'm totally novice with deployment and kubernetes things.

I'm using Kubernetes v1.5.2

-- Avv
circleci
docker
kubectl
kubernetes
kubernetes-helm

1 Answer

6/10/2019

You can try to assign more resources to your deployment definition for example, more CPU or RAM in some cases can increase the speed of a workload to boot up and be ready but maybe the workload didn't need it during the normal operations but in that case, you need to review your load process.

-- wolmi
Source: StackOverflow