I am running a cluster on Google Kubernetes Engine and I am currently trying to switch from using an Ingress with external load balancing (and NodePort services) to an ingress with container-native load balancing (and ClusterIP services) following this documentation: Container native load balancing
To communicate with my services I am using the following ingress configuration that used to work just fine when using NodePort services instead of ClusterIP:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: mw-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: mw-cluster-ip
networking.gke.io/managed-certificates: mw-certificate
kubernetes.io/ingress.allow-http: "false"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: billing-frontend-service
servicePort: 80
- path: /auth/api/*
backend:
serviceName: auth-service
servicePort: 8083
Now following the documentation, instead of using a readinessProbe as a part of the container deployment as a health check I switched to using ClusterIP services in combination with BackendConfig instead. For each deployment I am using a service like this:
apiVersion: v1
kind: Service
metadata:
labels:
app: auth
name: auth-service
namespace: default
annotations:
cloud.google.com/backend-config: '{"default": "auth-hc-config"}'
spec:
type: ClusterIP
selector:
app: auth
ports:
- port: 8083
protocol: TCP
targetPort: 8083
And a Backend config:
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: auth-hc-config
spec:
healthCheck:
checkIntervalSec: 10
port: 8083
type: http
requestPath: /auth/health
As a reference, this is what the readinessProbe used to look like before:
readinessProbe:
failureThreshold: 3
httpGet:
path: /auth/health
port: 8083
scheme: HTTP
periodSeconds: 10
Now to the actual problem. I deploy the containers and services first and they seem to startup just fine. The ingress however does not seem to pick up the health checks properly and shows this in the Cloud console:
Error during sync: error running backend syncing routine: error ensuring health check: googleapi: Error 400: Invalid value for field 'resource.httpHealthCheck': ''. HTTP healthCheck missing., invalid
The cluster as well as the node pool are running GKE version 1.17.6-gke.11 so the annotation cloud.google.com/neg: '{"ingress": true}'
is not necessary. I have checked and the service is annotated correctly:
Annotations: cloud.google.com/backend-config: {"default": "auth-hc-config"}
cloud.google.com/neg: {"ingress":true}
cloud.google.com/neg-status: {"network_endpoint_groups":{"8083":"k8s1-2078beeb-default-auth-service-8083-16a14039"},"zones":["europe-west3-b"]}
I have already tried to re-create the cluster and the node-pool with no effect. Any ideas on how to resolve this? Am I missing an additional health check somewhere?
I found my issue. Apparently the BackendConfig's type
attribute is case-sensitive. Once I changed it from http
to HTTP
it worked after I recreated the ingress.