I did a "deep dive" into that and managed to reproduce the situation on my GKE cluster, so now I can tell that there are a few things combined here.

A backend service happens to be returning Status 404 on the health check path of the Load Balancer.

There could be 2 options (it is not clear from the description you have provided).

• something like: "Error: Server Error The server encountered a temporary error and could not complete your request. Please try again in 30 seconds."

This one you are geting from LoadBalancer in case HealthCheck failed for pod. The official documentation on GKE Ingress object says that

a Service exposed through an Ingress must respond to health checks from the load balancer.

Any container that is the final destination of load-balanced traffic must do one of the following to indicate that it is healthy:

• Serve a response with an HTTP 200 status to GET requests on the / path.

• Configure an HTTP readiness probe. Serve a response with an HTTP 200 status to GET requests on the path specified by the readiness probe. The Service exposed through an Ingress must point to the same container port on which the readiness probe is enabled.

It is needed to fix HealthCheck handling. You can check Load balancer details by visiting GCP console - Network Services - Load Balancing.

• "404 Not Found -- nginx/1.17.6"

This one is clear. That is the response returned by endpoint myservice is sending request to. It looks like something is misconfigured there. My guess is that pod merely can't serve that request properly. Can be nginx web-server issue, etc. Please check the configuration to find out why pod can't serve the request.

While playing with the setup I have find an image that allows you to check if request has reached the pod and requests headers.

so it is possible to create a pod like:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    run: fake-web
  name: fake-default-knp
#  namespace: kube-system
spec:
  containers:
  - image: mendhak/http-https-echo
    imagePullPolicy: IfNotPresent
    name: fake-web
    ports:
    - containerPort: 8080
      protocol: TCP

to be able to see all the headers that were in incoming requests (kubectl logs -f fake-default-knp ).

When I browse to the Load Balancer's Static IP, my browser shows the 404 Error Message which the underlying Kubernetes Pod returned.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress1
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: myservice
          servicePort: 80

Upon creation of such an Ingress object, there will be at least 2 backends in GKE cluster. - the backend you have specified upon Ingress creation ( myservice one) - the default one (created upon cluster creation).

kubectl get pods -n kube-system -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP       
l7-default-backend-xyz     1/1     Running   0          20d   10.52.0.7

Please note that myservice serves only requests that have Host header set to example.com . The rest of requests are sent to "default backend" . That is the reason why you are receiving "default backend - 404" error message upon browsing to LoadBalancer's IP address.

Technically there is a default-http-backend service that has l7-default-backend-xyz as an EndPoint.

kubectl get svc -n kube-system -o wide 
NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE   SELECTOR
default-http-backend   NodePort    10.0.6.134    <none>        80:31806/TCP    20d   k8s-app=glbc

kubectl get ep -n kube-system
NAME                   ENDPOINTS       AGE
default-http-backend   10.52.0.7:8080  20d

Again, that's the "object" that returns the "default backend - 404" error for the requests with "Host" header not equal to the one you specified in Ingress.

Hope that it sheds a light on the issue :)

EDIT:

myservice serves only requests that have Host header set to example.com." So you are saying that requests go to the LB only when there is a host header?

Not exactly. The LB receives all the requests and passes requests in accordance to "Host" header value. Requests with example.com Host header are going to be served on myservice backend .

To put it simple the logic is like the following:

1. request arrives;
2. system checks the Host header (to determine user's backend)
3. request is served if there is a suitable user's backend ( according to the Ingress config) and that backend is healthy , otherwise "Error: Server Error The server encountered a temporary error and could not complete your request. Please try again in 30 seconds." is thrown if backend is in non-healthy state;
4. if request's Host header doesn't match any host in Ingress spec, request is sent to l7-default-backend-xyz backend (not the one that is mentioned in Ingress config). That backend replies with: "default backend - 404" error .

Hope that makes it clear.