I've follow the documentation about how to enable IAP on GKE.

I've:

1. configured the consent screen
2. Create OAuth credentials
3. Add the universal redirect URL
4. Add myself as IAP-secured Web App User

And write my deployment like this:

data:
  client_id: <my_id>
  client_secret: <my_secret>
kind: Secret
metadata:
  name: backend-iap-secret
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 3000
  selector:
    k8s-app: grafana
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: grafana
    spec:
      containers:
      - env:
        - name: GF_SERVER_HTTP_PORT
          value: "3000"
        image: docker.io/grafana/grafana:6.7.1
        name: grafana
        ports:
        - containerPort: 3000
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /api/health
            port: 3000
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: backend-config-iap
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: backend-iap-secret
---
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: monitoring-tls
spec:
  domains:
  - monitoring.foo.com
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    beta.cloud.google.com/backend-config: '{"default": "backend-config-iap"}'
    kubernetes.io/ingress.global-static-ip-name: monitoring
    networking.gke.io/managed-certificates: monitoring-tls
  name: grafana
spec:
  backend:
    serviceName: grafana
    servicePort: 443

When I look at my ingress I've this:

$ k describe ingress
Name:             grafana
[...]
Annotations:  beta.cloud.google.com/backend-config: {"default": "backend-config-iap"}
              ingress.kubernetes.io/backends: {"k8s-blabla":"HEALTHY"}
              [...] 
Events:       <none>
$

I can connect to the web page without any problem, the grafana is up and running, but I can also connect without being authenticated (witch is a problem).

So everything look fine, but IAP is not activated, why ?

The worst is that, if I enable it manualy it work but if I redo kubectl apply -f monitoring.yaml IAP is disabled.

What am I missing ?

Because my secret values are stored in secret manager (and retrieved at build time) I suspected my secret to have some glitches (spaces, \n, etc.) in them so I've add a script to test it:

gcloud compute backend-services update \
  --project=<my_project_id> \
  --global \
  $(kubectl get ingress grafana -o json | jq -r '.metadata.annotations."ingress.kubernetes.io/backends"' | jq -r 'keys[0]') \
  --iap=enabled,oauth2-client-id=$(gcloud --project="<my_project_id>" beta secrets versions access latest --secret=Monitoring_client_id),oauth2-client-secret=$(gcloud --project="<my_project_id>" beta secrets versions access latest --secret=Monitoring_secret)

And now IAP is properly enabled with the correct OAuth Client, so my secrets are "clean"

By the way, I also tried to rename secret variables like this (from client_id): * oauth_client_id * oauth-client-id * clientID (like in backend documentation )

I've also write the value in the backend like this:

kind: BackendConfig
metadata:
  name: backend-config-iap
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: backend-iap-secret
      clientID: <value>
      clientSecret: <value>

But doesn't work either.

Erratum:

The fact that the IAP is destroyed when I deploy again (after I enable it in web UI) is part of my deployment script in this test (I made a kubectl delete before).

But nevertheless, I can't enable IAP only with my backend configuration.

As suggested I've filed a bug report: https://issuetracker.google.com/issues/153475658

Solution given by Totem

Change given yaml with this:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: monitoring
    networking.gke.io/managed-certificates: monitoring-tls
  name: grafana
[...]
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  annotations:
    beta.cloud.google.com/backend-config: '{"default": "backend-config-iap"}'
[...]

The backend is associated with the service and not the Ingress... Now it Works !