Here is an example cluster setup that matches my production cluster:
Ingress rule abc.com > Service-AB > Pod-A, Pod-B > cluster-A, cluster-B > image-0.1.1
Ingress rule xyz.com > Service-CD > Pod-C, Pod-D > cluster-C, cluster-D > image-1.3.2
Imagine I need to update cluster1's image to v0.1.2
, to do this I can just change the deployment manifest and apply
it to perform a rolling update but before that I want to do an e2e test in production cluster to make sure the new image, behaves as expected there.
To accomplish this, I create a Pod Spec having the new image version together with a new service that selects only that new pod and an ingress rule to reach this new service. It looks like following,
Ingress rule abc.com/update > Service-Z > Pod-Z > cluster-Z > image-0.1.2
Now I point my e2e test to the endpoint abc.com/update
, once successful, I delete the new ingress rule, service and pod and follow the normal process of rolling update by updating the deployment with new container image version and performing an apply
.
Is it a correct way to do testing in production cluster, can there be a better (easier) way?
I think it varies depending on the technology that you are using. Given your case, which is using plain kubernetes. I would use a proxy in between to do the A/B testing. Here is the description:
Before upgrade:
Ingress rule abc.com -> svc-proxy -> pod-proxy --> svc-my-pod (v1.0.0)
After upgrade (A/B) testing:
Ingress rule abc.com -> svc-proxy -> pod-proxy --> svc-my-pod-1 (v1.0.0)
|
--> svc-my-pod-2 (v1.1.0)
Fully upgraded
Ingress rule abc.com -> svc-proxy -> pod-proxy -> my-pod (v1.1.0)
Note that in the A/B testing phase you would have to add an upstream with weights (if using nginx) to emulate the desired weighted policy.
upstream dynamic {
server pod-proxy-1 weight=2;
server pod-proxy-2 weight=4;
}
server {
location / {
proxy_pass http://dynamic;
}
}
Have a yaml with v1 called pod-v1. You would define its match label like the following yaml:
spec:
selector:
matchLabels:
app: pod-v1
Have a yaml with v2 called pod-v2. You would define its match label like the following yaml:
spec:
selector:
matchLabels:
app: pod-v2
Then you can easily change routing between the pods by changing the selector of a service. Like the following yaml:
apiVersion: v1
kind: Service
metadata:
name: pod
labels:
app: pod
spec:
ports:
- name: "8000"
port: 8000
targetPort: 8000
selector:
app: pod-v1
status:
loadBalancer: {}
Note that to route the traffic to v2 you would just change the selector to match the label pod-v2.
selector:
app: pod-v2