How to configure readinessProbe correctly for containers that do http to https redirects?

2/5/2019

We have an GKE ingress that is using the below frontend-service. The ingress terminates tls as well. We want to have http to https permanent redirects for any traffic that comes on http.

With the below configuration we have all working, and serving traffic on both http and https (without redirect).

The container used for the Deployment can be configured to rewrite http to https with --https-redirect flag. It also respect and trust the X-Forwarded-Proto header, and will consider it to be secure if the header value is set to https.

So the most reasonable configuration I can see for the readinessProbe would be the configuration below, but with the commented lines uncommented. However, as soon as we apply this version we never get into a healthy state, and instead the terminated domain configured with the Ingress returns with 502 and never recovers.

So what is wrong with the below approach? I have tested using port-forwarding both the pod and the service, and they both return 301 if I do not provide the X-Forwarded-Proto header, and return 200 if I provide the X-Forwarded-Proto header with https value.

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  type: NodePort
  ports:
    - port: 8080
  selector:
    app: frontend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - image: eu.gcr.io/someproject/frontend:master
        imagePullPolicy: Always
        # args:
        # - '--https-redirect'
        name: frontend
        resources:
          limits:
            memory: 1Gi
            cpu: '0.5'
        ports:
        - containerPort: 8080
          name: frontend
        readinessProbe:
          httpGet:
            path: /_readinessProbe
            port: 8080
            # httpHeaders:
            # - name: X-Forwarded-Proto
            #   value: https
-- pjotr_dolphin
google-kubernetes-engine
kubernetes
kubernetes-health-check
kubernetes-ingress

1 Answer

2/5/2019

The GCP Health Check is very picky about the HTTP response codes it gets back. It must be a 200, and not a redirect. If in the configuration you have posted, the NLB gets a 301/302 response from your server. it will then mark your backend as unhealthy as this is not a 200 response. If the health check is sending HTTP without the X-Forwarded-Proto header, this is likely.

You can check this by inspecting the kubectl logs of your deployment's pods.

My previous answer may be useful if you move to an HTTPS health check, in an attempt to remedy this.


From GKE documentation: You will need to put an annotation on your Service definition that tells GKE to use HTTPS for the health check. Otherwise it will try sending HTTP and get confused.

kind: Service
metadata:
  name: my-service-3
  annotations:
    cloud.google.com/app-protocols: '{"my-https-port":"HTTPS","my-http-port":"HTTP"}'
spec:
  type: NodePort
  selector:
    app: metrics
    department: sales
  ports:
  - name: my-https-port
    port: 443
    targetPort: 8443
  - name: my-http-port
    port: 80
    targetPort: 50001

I haven't used the latest syntax, but this used to work for me.

However this was so clunky to use I ended up going over to Istio and getting that to do all the HTTPS termination. That's no small undertaking however, but it you're thinking of using cert-manager/Let's Encrypt it might be worth exploring.

-- Paul Annetts
Source: StackOverflow