Goal

I'm trying to setup a

Cloud LB -> GKE [istio-gateway -> my-service]

This was working before, however, I have to recreate the cluster 2 days ago and run into this problem. Maybe some version change?

This is my ingress manifest file

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "my-dev-ingress"
  namespace: "istio-system"
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "my-dev-gclb-ip"
    ingress.gcp.kubernetes.io/pre-shared-cert: "my-dev-cluster-cert-05"
    kubernetes.io/ingress.allow-http: "false"
spec:
  backend:
    serviceName: "istio-ingressgateway"
    servicePort: 80

Problem

The health check issue by the Cloud LB failed. The backend service created by the Ingress create a /:80 default health check.

What I have tried

1) I tried to set the health check generated by the gke ingress to point to the istio-gateway StatusPort port 15020 in the Backend config console. Then the health check passed for a bit until the backend config revert itself to use the original /:80 healthcheck that it created. I even tried to delete the healthcheck that it created and it just create another one.

2) I also tried using the istio-virtual service to route the healthcheck to 15020 port as shown here with out much success.

3) I also tried just route everything in the virtual-service the healthcheck port

  hosts:
    - "*"
  gateways:
    - my-web-gateway
  http:
    - match:
        - method:
            exact: GET
          uri:
            exact: /
      route:
        - destination:
            host: istio-ingress.gke-system.svc.cluster.local
            port:
              number: 15020

4) Most of the search result I found say that setting readinessProbe in the deployment should tell the ingress to set the proper health check. However, all of my service are under the istio-gateway and I can't really do the same.

I'm very lost right now and will really appreciate it if anyone could point me to the right direction. Thanks