Intro to the issue
I have a gRPC service that is running in GKE. I'm using Cloud Endpoints and have an ESP (Extensible Service Proxy) in front of it that I want to use to handle authentication (JWT tokens).

This sounds like a really great way to integrate all these tools together, but for the past 2 days I haven't been able to get this working. I can connect and make calls perfectly when running locally, but going through the ESP has proven to be pretty difficult to setup.

Note: I eventually want to open up an http2 port, but my focus right now is getting browser support so I'm just opening the http port

The K8s deployment and service config
My service is running and working, and I am deploying it with this manifest

apiVersion: v1
kind: Service
metadata:
  name: esp-grpc-environment
spec:
  ports:
  # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
  - port: 80
    targetPort: 9090
    protocol: TCP
    name: http2
  selector:
    app: esp-grpc-environment
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: esp-grpc-environment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: esp-grpc-environment
    spec:
      containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1.16.0
        args: [
          "--http_port=9090",
          "--service=environment.endpoints.olamai-d64a7.cloud.goog",
          "--rollout_strategy=managed",
          "--backend=grpc://127.0.0.1:8000",
          "--cors_preset=basic",
          "--cors_allow_headers=Keep-Alive,User-Agent,Cache-Control,Content-Type,Content-Transfer-Encoding,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Grpc-Timeout,Authorization,authorization",
          "--cors_expose_headers=grpc-status,grpc-message,authorization"
        ]
        ports:
          - containerPort: 9090
      - name: environment
        image: terrariumai/environment:0.0.1
        imagePullPolicy: Always
        ports:
          - containerPort: 8000

So I was originally just using my own nginx proxy but decided I wanted the features of Endpoints. With this and a lot of errors, I got all of the Cors arguments for the ESP.

And here is the config with the auth settings

type: google.api.Service
config_version: 3

name: environment.endpoints.<project id>.cloud.goog
title: Environment gRPC API

apis:
  - name: endpoints.terrariumai.environment.Environment

authentication:
  providers:
    - id: firebase
      jwks_uri: https://www.googleapis.com/service_accounts/v1/metadata/x509/securetoken@system.gserviceaccount.com
      issuer: https://securetoken.google.com/<project-id>
  rules:
    - selector: "*"
      requirements:
        - provider_id: firebase

usage:
  rules:
    - selector: endpoints.terrariumai.environment.Environment.CreateEntity
      allow_unregistered_calls: true

Making calls to the service with Node/React
So once this is running and deployed, I use my React app to hit the service like this

import {
  CreateEntityRequest
} from "../api/environment_pb";

this.props.firebase
      .auth()
      .currentUser.getIdToken(/* forceRefresh */ true)
      .then(function(idToken) {
        var service = new EnvironmentClient(addr, null, null);
        var request = new CreateEntityRequest();
        var metadata = {
          authorization: `Bearer ${idToken}`
        };
        console.log(idToken);
        service.createEntity(request, metadata, (err, resp) => {
          if (err) {
            console.log("Got error: ", err);
          }
          console.log("Resp: ", resp);
        });
      })

And from this, I get this error

POST http://<external-ip>/endpoints.terrariumai.environment.Environment/CreateEntity 403 (Forbidden)

But at this point I'm so confused. Here are the log from the ESP pod (which I'm having trouble reading)

[09/Jul/2019:20:40:49 +0000] "POST /endpoints.terrariumai.environment.Environment/CreateEntity HTTP/1.1" 403 210 "http://localhost:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

To me, it seems like the ESP is refusing the calls and saying they are forbidden (maybe not authenticated??). But at this point I'm stumped. I really don't know where to look further. I would appreciate any help and can provide any more information if it's necessary!