I'm trying a simple ingress in gke. Following the example from https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress

the pods are up and running, services are active. When I create ingress I'm getting

Events:
  Type     Reason  Age                  From                     Message
  ----     ------  ----                 ----                     -------
  Normal   ADD     48m                   loadbalancer-controller  default/my-ingress
  Warning  Sync    2m32s (x25 over 48m)  loadbalancer-controller  Error during sync: Error running backend syncing routine: googleapi: got HTTP response code 404 with body: Not Found

I can't find the source of the problem. Any suggestion of where to look?

I have checked cluster add-ons and permissions httpLoadBalancing enabled

  - https://www.googleapis.com/auth/compute
  - https://www.googleapis.com/auth/devstorage.read_only
  - https://www.googleapis.com/auth/logging.write
  - https://www.googleapis.com/auth/monitoring
  - https://www.googleapis.com/auth/servicecontrol
  - https://www.googleapis.com/auth/service.management.readonly
  - https://www.googleapis.com/auth/trace.append

NAME                                          READY   STATUS    RESTARTS   AGE
hello-kubernetes-deployment-f6cb6cf4f-kszd9   1/1     Running   0          1h
hello-kubernetes-deployment-f6cb6cf4f-lw49t   1/1     Running   0          1h
hello-kubernetes-deployment-f6cb6cf4f-qqgxs   1/1     Running   0          1h
hello-world-deployment-5cfbc486f-4c2bm        1/1     Running   0          1h
hello-world-deployment-5cfbc486f-dmcqf        1/1     Running   0          1h
hello-world-deployment-5cfbc486f-rnpcc        1/1     Running   0          1h

Name:                     hello-world
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"hello-world","namespace":"default"},"spec":{"ports":[{"port":6000...
Selector:                 department=world,greeting=hello
Type:                     NodePort
IP:                       10.59.254.88
Port:                     <unset>  60000/TCP
TargetPort:               50000/TCP
NodePort:                 <unset>  30418/TCP
Endpoints:                10.56.2.7:50000,10.56.3.6:50000,10.56.6.4:50000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Name:                     hello-kubernetes
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"hello-kubernetes","namespace":"default"},"spec":{"ports":[{"port"...
Selector:                 department=kubernetes,greeting=hello
Type:                     NodePort
IP:                       10.59.251.189
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  32464/TCP
Endpoints:                10.56.2.6:8080,10.56.6.3:8080,10.56.8.6:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Name:             my-ingress
Namespace:        default
Address:
Default backend:  default-http-backend:80 (10.56.0.9:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /*      hello-world:60000 (<none>)
        /kube   hello-kubernetes:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"gce"},"name":"my-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"hello-world","servicePort":60000},"path":"/*"},{"backend":{"serviceName":"hello-kubernetes","servicePort":80},"path":"/kube"}]}}]}}

  kubernetes.io/ingress.class:  gce
Events:
  Type     Reason  Age                  From                     Message
  ----     ------  ----                 ----                     -------
  Normal   ADD     107s                 loadbalancer-controller  default/my-ingress
  Warning  Sync    66s (x15 over 107s)  loadbalancer-controller  Error during sync: Error running backend syncing routine: googleapi: got HTTP response code 404 with body: Not Found

Pulumi Cluster Config

                {
                    "name": "test-cluster",
                    "region": "europe-west4",
                    "addonsConfig": {
                        "httpLoadBalancing": {
                            "disabled": false
                        },
                        "kubernetesDashboard": {
                            "disabled": false
                        }
                    },
                    "ipAllocationPolicy": {},
                    "pools": [
                        {
                            "name": "default-pool",
                            "initialNodeCount": 1,
                            "nodeConfig": {
                                "oauthScopes": [
                                    "https://www.googleapis.com/auth/compute",
                                    "https://www.googleapis.com/auth/devstorage.read_only",
                                    "https://www.googleapis.com/auth/service.management",
                                    "https://www.googleapis.com/auth/servicecontrol",
                                    "https://www.googleapis.com/auth/logging.write",
                                    "https://www.googleapis.com/auth/monitoring",
                                    "https://www.googleapis.com/auth/trace.append",
                                    "https://www.googleapis.com/auth/cloud-platform"
                                ],
                                "machineType": "n1-standard-1",
                                "labels": {
                                    "pool": "api-zero"
                                }
                            },
                            "management": {
                                "autoUpgrade": false,
                                "autoRepair": true
                            },
                            "autoscaling": {
                                "minNodeCount": 1,
                                "maxNodeCount": 20
                            }
                        },
                        {
                            "name": "outbound",
                            "initialNodeCount": 2,
                            "nodeConfig": {
                                "machineType": "custom-1-1024",
                                "oauthScopes": [
                                    "https://www.googleapis.com/auth/compute",
                                    "https://www.googleapis.com/auth/devstorage.read_only",
                                    "https://www.googleapis.com/auth/service.management",
                                    "https://www.googleapis.com/auth/servicecontrol",
                                    "https://www.googleapis.com/auth/logging.write",
                                    "https://www.googleapis.com/auth/monitoring",
                                    "https://www.googleapis.com/auth/trace.append",
                                    "https://www.googleapis.com/auth/cloud-platform"
                                ],
                                "labels": {
                                    "pool": "outbound"
                                }
                            },
                            "management": {
                                "autoUpgrade": false,
                                "autoRepair": true
                            }
                        }