We have an GKE ingress that is using the below frontend-service. The ingress terminates tls as well. We want to have http to https permanent redirects for any traffic that comes on http.
With the below configuration we have all working, and serving traffic on both http and https (without redirect).
The container used for the Deployment can be configured to rewrite http to https with --https-redirect flag. It also respect and trust the X-Forwarded-Proto header, and will consider it to be secure if the header value is set to https.
So the most reasonable configuration I can see for the readinessProbe would be the configuration below, but with the commented lines uncommented. However, as soon as we apply this version we never get into a healthy state, and instead the terminated domain configured with the Ingress returns with 502 and never recovers.
So what is wrong with the below approach? I have tested using port-forwarding both the pod and the service, and they both return 301 if I do not provide the X-Forwarded-Proto header, and return 200 if I provide the X-Forwarded-Proto header with https value.
apiVersion: v1
kind: Service
metadata:
name: frontend-service
spec:
type: NodePort
ports:
- port: 8080
selector:
app: frontend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- image: eu.gcr.io/someproject/frontend:master
imagePullPolicy: Always
# args:
# - '--https-redirect'
name: frontend
resources:
limits:
memory: 1Gi
cpu: '0.5'
ports:
- containerPort: 8080
name: frontend
readinessProbe:
httpGet:
path: /_readinessProbe
port: 8080
# httpHeaders:
# - name: X-Forwarded-Proto
# value: https
The GCP Health Check is very picky about the HTTP response codes it gets back. It must be a 200, and not a redirect. If in the configuration you have posted, the NLB gets a 301/302 response from your server. it will then mark your backend as unhealthy as this is not a 200 response. If the health check is sending HTTP without the X-Forwarded-Proto header, this is likely.
You can check this by inspecting the kubectl logs of your deployment's pods.
My previous answer may be useful if you move to an HTTPS health check, in an attempt to remedy this.
From GKE documentation: You will need to put an annotation on your Service definition that tells GKE to use HTTPS for the health check. Otherwise it will try sending HTTP and get confused.
kind: Service
metadata:
name: my-service-3
annotations:
cloud.google.com/app-protocols: '{"my-https-port":"HTTPS","my-http-port":"HTTP"}'
spec:
type: NodePort
selector:
app: metrics
department: sales
ports:
- name: my-https-port
port: 443
targetPort: 8443
- name: my-http-port
port: 80
targetPort: 50001
I haven't used the latest syntax, but this used to work for me.
However this was so clunky to use I ended up going over to Istio and getting that to do all the HTTPS termination. That's no small undertaking however, but it you're thinking of using cert-manager/Let's Encrypt it might be worth exploring.