K
Q

Running integration/e2e tests on top of a Kubernetes stack

10/24/2019

I’ve been digging a bit into the way people run integration and e2e tests in the context of Kubernetes and have been quite disappointed by the lack of documentation and feedbacks. I know there are amazing tools such as kind or minikube that allow to run resources locally. But in the context of a CI, and with a bunch of services, it does not seem to be a good fit, for obvious resources reasons. I think there are great opportunities with running tests for:

  • Validating manifests or helm charts
  • Validating the well behaving of a component as part of a bigger whole
  • Validating the global behaviour of a product

The point here is not really about the testing framework but more about the environment on top of which the tests could be run.

Do you share my thought? Have you ever experienced running such kind of tests? Do you have any feedbacks or insights about it?

Thanks a lot

-- vesna
continuous-integration
kubernetes
testing

Similar Questions

HPA scale down not happening properly
Production ready Python apps on Kubernetes
container_fs_ metrics are coming zero for all pods in prometheus operator
Kubernetes ingress websockets connection issue

1 Answer

10/24/2019

Interesting question and something that I have worked on over the last couple of months for my current employer. Essentially we ship a product as docker images with manifests. When writing e2e tests I want to run the product as close to the customer environment as possible.

Essentially to solve this we have built scripts that interact with our standard cloud provider (GCloud) to create a cluster, deploy the product and then run the tests against it.

For the major cloud providers this is not a difficult tasks but can be time consuming. There are a couple of things that we have learnt the hard way to keep in mind while developing the tests.

  1. Concurrency, this may sound obvious but do think about the number of concurrent builds your CI can run.
  2. Latency from the cloud, don't assume that you will get an instant response to every command that you run in the cloud. Also think about the timeouts. If you bring up a product with lots of pods and services what is the acceptable start up time?
  3. Errors causing build failures, this is an interesting one. We have seen errors in the build due to network errors when communicating with our test deployment. These are nearly always transitive. It is best to avoid these making the build fail.

One thing to look at is GitLab are providing some documentation on how to build and test images in their CI pipeline.

-- James Wilson
Source: StackOverflow