GCE health check not working with ingress nginx controller

10/30/2020

I am installing nginx ingress controller (k8s.gcr.io/ingress-nginx/controller:v0.40.2) on a GKE cluster v1.17.12-gke.1504 with 3 nodes.

The ingress-nginx-controller is deployed as a Deployment; I don't want to deploy it as a DaemonSet because I'm going to have a cluster with many nodes in the future and it's going to be a waste of resources.

The creation of the kubernetes resources completes fine and the load balancer is created in GCE but the associated health check appears as failed.

I have accessed each of the cluster nodes to check the endpoint of the healt check and I see that it only works on the node where the ingress-nginx-controller pod has been created.

In the node where the pod has been raised it returns a 200:

$ curl localhost:32203/healthz -v
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 32203 (#0)
> GET /healthz HTTP/1.1
> Host: localhost:32203
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 30 Oct 2020 09:03:42 GMT
< Content-Length: 111
<
{
        "service": {
                "namespace": "ingress-nginx",
                "name": "ingress-nginx-controller"
        },
        "localEndpoints": 1
* Connection #0 to host localhost left intact
}* Closing connection 0

But on nodes where the pod is not up it returns a 503:

~ $ curl localhost:32203/healthz -v
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 32203 (#0)
> GET /healthz HTTP/1.1
> Host: localhost:32203
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json
< Date: Fri, 30 Oct 2020 09:03:20 GMT
< Content-Length: 111
<
{
        "service": {
                "namespace": "ingress-nginx",
                "name": "ingress-nginx-controller"
        },
        "localEndpoints": 0
* Connection #0 to host localhost left intact
}* Closing connection 0

How can I get the health check to work without having to lift a pod from the ingress-nginx-controller on each node?

-- Diego
google-kubernetes-engine
kubernetes
kubernetes-health-check

1 Answer

10/31/2020

the problem was that I had configured externalTrafficPolicy: "Local" in the service.

https://kubernetes.io/docs/tutorials/services/source-ip/

-- Diego
Source: StackOverflow