is my method of performing production testing before rolling update in kubernetes correct?

10/9/2019

Here is an example cluster setup that matches my production cluster:

Ingress rule abc.com > Service-AB > Pod-A, Pod-B > cluster-A, cluster-B > image-0.1.1

Ingress rule xyz.com > Service-CD > Pod-C, Pod-D > cluster-C, cluster-D > image-1.3.2

Imagine I need to update cluster1's image to v0.1.2, to do this I can just change the deployment manifest and apply it to perform a rolling update but before that I want to do an e2e test in production cluster to make sure the new image, behaves as expected there.

To accomplish this, I create a Pod Spec having the new image version together with a new service that selects only that new pod and an ingress rule to reach this new service. It looks like following,

Ingress rule abc.com/update > Service-Z > Pod-Z > cluster-Z > image-0.1.2

Now I point my e2e test to the endpoint abc.com/update, once successful, I delete the new ingress rule, service and pod and follow the normal process of rolling update by updating the deployment with new container image version and performing an apply.

Is it a correct way to do testing in production cluster, can there be a better (easier) way?

-- p_champ
kubernetes

1 Answer

10/9/2019

I think it varies depending on the technology that you are using. Given your case, which is using plain kubernetes. I would use a proxy in between to do the A/B testing. Here is the description:

Before upgrade:
Ingress rule abc.com -> svc-proxy -> pod-proxy --> svc-my-pod (v1.0.0)

After upgrade (A/B) testing:
Ingress rule abc.com -> svc-proxy -> pod-proxy --> svc-my-pod-1 (v1.0.0)
                      |
                      --> svc-my-pod-2 (v1.1.0)

Fully upgraded
Ingress rule abc.com -> svc-proxy -> pod-proxy -> my-pod (v1.1.0)

Note that in the A/B testing phase you would have to add an upstream with weights (if using nginx) to emulate the desired weighted policy.

upstream dynamic {
    server pod-proxy-1      weight=2;
    server pod-proxy-2      weight=4;
}

server {
    location / {
        proxy_pass http://dynamic;
    }
}

Using services and endpoints to switch between pods

Have a yaml with v1 called pod-v1. You would define its match label like the following yaml:

spec:
  selector:
    matchLabels:
      app: pod-v1

Have a yaml with v2 called pod-v2. You would define its match label like the following yaml:

spec:
  selector:
    matchLabels:
      app: pod-v2

Then you can easily change routing between the pods by changing the selector of a service. Like the following yaml:

apiVersion: v1
kind: Service
metadata:
  name: pod
  labels:
    app: pod
spec:
  ports:
  - name: "8000"
    port: 8000
    targetPort: 8000
  selector:
    app: pod-v1
status:
  loadBalancer: {}

Note that to route the traffic to v2 you would just change the selector to match the label pod-v2.

selector:
  app: pod-v2
-- Rodrigo Loza
Source: StackOverflow