I've been working on a gRPC service that requires the ability to receive calls from a browser being authenticated with Firebase. I got everything working up until this 409 error but can't seem to find any more info on it.

I'll try to provide as much info as I can right now.

The Code

Here is my k8s manifest

apiVersion: v1
kind: Service
metadata:
  name: esp-grpc-environment
spec:
  ports:
  # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
  - port: 80
    targetPort: 9090
    protocol: TCP
    name: http2
  selector:
    app: esp-grpc-environment
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: esp-grpc-environment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: esp-grpc-environment
    spec:
      containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1
        args: [
          "--http_port=9090",
          "--service=environment.endpoints.<project_id>.cloud.goog",
          "--rollout_strategy=managed",
          "--backend=grpc://127.0.0.1:8000",
          "--cors_preset=basic",
          "--cors_allow_headers=Keep-Alive,User-Agent,Cache-Control,Content-Type,Content-Transfer-Encoding,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Grpc-Timeout,Authorization,authorization",
          "--cors_expose_headers=grpc-status,grpc-message,authorization",
          "--enable_debug"
        ]
        ports:
          - containerPort: 9090
      - name: environment
        image: terrariumai/environment:0.0.1
        imagePullPolicy: Always
        ports:
          - containerPort: 8000

And my endpoints config

type: google.api.Service
config_version: 3

name: environment.endpoints.<project_id>.cloud.goog
title: Environment gRPC API

apis:
  - name: endpoints.terrariumai.environment.Environment

authentication:
  providers:
    - id: firebase
      jwks_uri: https://www.googleapis.com/service_accounts/v1/metadata/x509/securetoken@system.gserviceaccount.com
      issuer: https://securetoken.google.com/<project_id>
  rules:
    - selector: "*"
      requirements:
        - provider_id: firebase

usage:
  rules:
    - selector: endpoints.terrariumai.environment.Environment.CreateEntity
      allow_unregistered_calls: true

Here is a link to my service config

Service Config

And this is how I am calling the API

import {
CreateEntityRequest
} from "../api/environment_pb";

this.props.firebase
      .auth()
      .currentUser.getIdToken(/* forceRefresh */ true)
      .then(function(idToken) {
        var service = new EnvironmentClient(addr, null, null);
        var request = new CreateEntityRequest();
        var metadata = {
          authorization: `Bearer ${idToken}`
        };
        console.log(idToken);
        service.createEntity(request, metadata, (err, resp) => {
          if (err) {
            console.log("Got error: ", err);
          }
          console.log("Resp: ", resp);
        });
      })

The errors

So at this point I get this error in the browser

POST http://<external-ip>/endpoints.terrariumai.environment.Environment/CreateEntity 403 (Forbidden) 

And here is the relevant content (but not all of it, I think this should be enough) from /var/log/nginx/error.log

2019/07/11 00:06:44 [debug] 9#9: *8 HTTP/1.1 403 Forbidden
Server: nginx
Date: Thu, 11 Jul 2019 00:06:44 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
WWW-Authenticate: Bearer, error="invalid_token"
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
Access-Control-Allow-Headers: Keep-Alive,User-Agent,Cache-Control,Content-Type,Content-Transfer-Encoding,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Grpc-Timeout,Authorization,authorization
Access-Control-Expose-Headers: grpc-status,grpc-message,authorization

2019/07/11 00:06:44 [debug] 9#9: *8 write new buf t:1 f:0 000056541D4E9778, pos 000056541D4E9778, size: 618 file: 0, size: 0
2019/07/11 00:06:44 [debug] 9#9: *8 http write filter: l:0 f:0 s:618
2019/07/11 00:06:44 [debug] 9#9: *8 http output filter "/endpoints.terrariumai.environment.Environment/CreateEntity?"
2019/07/11 00:06:44 [debug] 9#9: *8 ESP error message: JWT validation failed: Audience not allowed
2019/07/11 00:06:44 [debug] 9#9: *8 send error response: {
 "code": 7,
 "message": "JWT validation failed: Audience not allowed",
 "details": [
  {
   "@type": "type.googleapis.com/google.rpc.DebugInfo",
   "stackEntries": [],
   "detail": "auth"
  }
 ]
}

I'm really not sure how to further debug this. I tried looking this up and the only documentation I can find says that maybe Firebase is setting the incorrect "aud" value in the header?? Any insight would be appreciated!