K
Q

Staging and production routes conflicting withone another

4/10/2020

Yesterday I was testing having a production and staging deployment running at the same time. The results I got were erratic in that sometimes you'd go to the staging URL and it would load the production. If you refreshed the page, it would switch between the two randomly.

I won't have time to test it until this weekend because QA is going on, and this issue disrupted it from happening. I killed the production deployment and removed the routes from my ingress.yaml so QA could continue without issue.

At any rate, my ingress.yaml configuration was likely the cause, so wanted to share it to see what caused this behavior:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.org/client-max-body-size: "500m"
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: ingress-service
  namespace: default
spec:
  tls:
    - hosts:
        - domain.com
        - www.domain.com
        - staging.domain.com
      secretName: tls-domain-com
  rules:
    - host: domain.com
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-cluster-ip-service-prod
              servicePort: 3000
          - path: /admin/?(.*)
            backend:
              serviceName: admin-cluster-ip-service-prod
              servicePort: 4000
          - path: /api/?(.*)
            backend:
              serviceName: api-cluster-ip-service-prod
              servicePort: 5000
    - host: www.domain.com
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-cluster-ip-service-prod
              servicePort: 3000
          - path: /admin/?(.*)
            backend:
              serviceName: admin-cluster-ip-service-prod
              servicePort: 4000
          - path: /api/?(.*)
            backend:
              serviceName: api-cluster-ip-service-prod
              servicePort: 5000
    - host: staging.domain.com
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-cluster-ip-service-staging
              servicePort: 3000
          - path: /admin/?(.*)
            backend:
              serviceName: admin-cluster-ip-service-staging
              servicePort: 4000
          - path: /api/?(.*)
            backend:
              serviceName: api-cluster-ip-service-staging
              servicePort: 5000

I have feeling it is one of the following:

  • Having the domain.com before the others and that it should be at the end
  • Now that I'm looking at it again, they are at the same IP and using the same ports, so the ports need to be changed
  • If I want to leave the ports the same, I'd need to deploy another ingress controller having one for staging and one for production and follow these instructions

At any rate, can anyone confirm?

EDIT

Adding .yaml:

# client-staging.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment-staging
spec:
  replicas: 3
  selector:
    matchLabels:
      component: client
  template:
    metadata:
      labels:
        component: client
    spec:
      containers:
        - name: client
          image: testappacr.azurecr.io/test-app-client
          ports:
            - containerPort: 3000
          env:
          - name: DOMAIN
            valueFrom:
              secretKeyRef:
                name: test-app-staging-secrets
                key: DOMAIN 
---
apiVersion: v1
kind: Service
metadata:
  name: client-cluster-ip-service-staging
spec:
  type: ClusterIP
  selector:
    component: client
  ports:
    - port: 3000
      targetPort: 3000

# client-prod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment-prod
spec:
  replicas: 3
  selector:
    matchLabels:
      component: client
  template:
    metadata:
      labels:
        component: client
    spec:
      containers:
        - name: client
          image: testappacr.azurecr.io/test-app-client
          ports:
            - containerPort: 3000
          env:
          - name: DOMAIN
            valueFrom:
              secretKeyRef:
                name: test-app-prod-secrets
                key: DOMAIN 

---
apiVersion: v1
kind: Service
metadata:
  name: client-cluster-ip-service-prod
spec:
  type: ClusterIP
  selector:
    component: client
  ports:
    - port: 3000
      targetPort: 3000
-- eox.dev
kubernetes
nginx-ingress

Similar Questions

Name resolution from windows pods does not work on 1.18.1
Issue in setting up KUBECTL on Windows 10 Home
Install Helm on Virtual Box
Minikube driver=podman, has anyone been able to get it to work?

1 Answer

4/10/2020

Without seeing the deployments yaml descriptors and by the description of the problem the most probable cause is that both deployments are getting in the service loadbalancers endpoints, so you must change the selectors and add something like: env: prod and env: staging and add those to each deployment.

For checking if this is the issue run a kubectl describe service for each service and check the endpoints.

If not, let me know the output and i can help you debug further.

EDIT: Changes in files after posting them:

Production

Service:

spec:
  type: ClusterIP
  selector:
    component: client
    environment: production

Deployment:

  replicas: 3
  selector:
    matchLabels:
      component: client
      environment: production
  template:
    metadata:
      labels:
        component: client
        environment: production

Staging

Service:

spec:
  type: ClusterIP
  selector:
    component: client
    environment: staging

Deployment:

  replicas: 3
  selector:
    matchLabels:
      component: client
      environment: staging
  template:
    metadata:
      labels:
        component: client
        environment: staging
-- paltaa
Source: StackOverflow