K
Q

Ambassador external authentication set-up returns 404

4/12/2019

I am using Ambassador as the API gateway and am trying to get the external auth to work. I have deployed the auth-service with the corresponding Ambassador annotations

...
metadata:
  name: auth-service-svc
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind:  AuthService
      name:  authentication
      proto: http
      path_prefix: "/api"
      auth_service:  auth-service-svc:8080    
spec:
  ports:
    - port: 8080
      targetPort: 8080
      name: http
  selector:
    app: auth-service
  type: ClusterIP

When I do a curl to initiate a Get request against http://[hostname]/api, I expect the auth-service to be called. However this doesn't seem to happen. When I look at the ambassador logs I see the following message:

[2019-04-12 15:10:42.977][75][debug][main] [source/server/connection_handler_impl.cc:257] [C20117] new connection
[2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:243] [C20117] new stream
[2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:580] [C20117][S12856858676043764009] request headers complete (end_stream=true):
':authority', '<some-prefix>.elb.amazonaws.com'
':path', '/api/'
':method', 'GET'
'user-agent', 'curl/7.61.0'
'accept', '*/*'

[2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:1037] [C20117][S12856858676043764009] request end stream
[2019-04-12 15:10:42.977][75][debug][router] [source/common/router/router.cc:277] [C20117][S12856858676043764009] no cluster match for URL '/api/'
[2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:1278] [C20117][S12856858676043764009] encoding headers via codec (end_stream=true):
':status', '404'
'date', 'Fri, 12 Apr 2019 15:10:42 GMT'
'server', 'envoy'

[2019-04-12 15:10:43.013][75][debug][connection] [source/common/network/connection_impl.cc:502] [C20117] remote close
[2019-04-12 15:10:43.013][75][debug][connection] [source/common/network/connection_impl.cc:183] [C20117] closing socket: 0

Here are the logs (truncated) when the auth-service is deleted and re-created. I have tried to show the logs that capture the update of the envoy config.

...
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: copying configuration from http://localhost:34525/api/snapshot/40 to /ambassador/snapshots/snapshot-tmp.yaml
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://auth-service-svc:8080 is None
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://127.0.0.1:8877 is None
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://127.0.0.1:8877 is None
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://127.0.0.1:8877 is None
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: extauth: server_uri http://api
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: listen_ports ['80']
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: -global-: NOTICE: Ambassador 0.60 will default to listening on port 8080 for HTTP. You will need to change your configuration to continue using port 80.
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: successfully validated the resulting envoy configuration, continuing...
....

[2019-04-12 15:20:59.782][65][debug][config] [source/server/listener_manager_impl.cc:56]     name: envoy.http_connection_manager
[2019-04-12 15:20:59.783][65][debug][config] [source/server/listener_manager_impl.cc:59]   config: {"use_remote_address":true,"access_log":[{"config":{"path":"/dev/fd/1","format":"ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"},"name":"envoy.file_access_log"}],"xff_num_trusted_hops":0,"normalize_path":true,"route_config":{"virtual_hosts":[{"name":"backend","routes":[{"route":{"priority":null,"prefix_rewrite":"/ambassador/v0/check_ready","timeout":"10.000s","weighted_clusters":{"clusters":[{"name":"cluster_127_0_0_1_8877","weight":100}]}},"match":{"prefix":"/ambassador/v0/check_ready","case_sensitive":true}},{"match":{"case_sensitive":true,"prefix":"/ambassador/v0/check_alive"},"route":{"priority":null,"prefix_rewrite":"/ambassador/v0/check_alive","timeout":"10.000s","weighted_clusters":{"clusters":[{"name":"cluster_127_0_0_1_8877","weight":100}]}}},{"match":{"case_sensitive":true,"prefix":"/ambassador/v0/"},"route":{"priority":null,"weighted_clusters":{"clusters":[{"weight":100,"name":"cluster_127_0_0_1_8877"}]},"timeout":"10.000s","prefix_rewrite":"/ambassador/v0/"}},{"match":{"case_sensitive":true,"prefix":"/leads/"},"route":{"timeout":"3.000s","prefix_rewrite":"/","weighted_clusters":{"clusters":[{"weight":100,"name":"cluster_lead_service_svc"}]},"priority":null}}],"domains":["*"]}]},"http_filters":[{"config":{"http_service":{"server_uri":{"timeout":"5.000s","uri":"http://api","cluster":"cluster_extauth_auth_service_svc_8080"},"authorization_request":{"allowed_headers":{"patterns":[{"exact":"x-forwarded-proto"},{"exact":"cookie"},{"exact":"user-agent"},{"exact":"proxy-authorization"},{"exact":"from"},{"exact":"authorization"},{"exact":"x-forwarded-for"},{"exact":"x-forwarded-host"}]}},"path_prefix":"/api","authorization_response":{"allowed_client_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]},"allowed_upstream_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]}}}},"name":"envoy.ext_authz"},{"name":"envoy.cors"},{"name":"envoy.router"}],"stat_prefix":"ingress_http"}
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: configuration updated from snapshot 40
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Scout reports {"latest_version": "0.53.1", "application": "ambassador", "cached": true, "timestamp": 1555081685.663466}
2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Scout notices: [{"level": "DEBUG", "message": "Returning cached result"}]
[2019-04-12 15:20:59.799][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:302]     http filter #0
[2019-04-12 15:20:59.799][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:303]       name: envoy.ext_authz
[2019-04-12 15:20:59.800][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:307]     config: {"http_service":{"authorization_response":{"allowed_upstream_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]},"allowed_client_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]}},"server_uri":{"timeout":"5.000s","uri":"http://api","cluster":"cluster_extauth_auth_service_svc_8080"},"authorization_request":{"allowed_headers":{"patterns":[{"exact":"x-forwarded-proto"},{"exact":"cookie"},{"exact":"user-agent"},{"exact":"proxy-authorization"},{"exact":"from"},{"exact":"authorization"},{"exact":"x-forwarded-for"},{"exact":"x-forwarded-host"}]}},"path_prefix":"/api"}}
[2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:302]     http filter #1
[2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:303]       name: envoy.cors
[2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:307]     config: {}
[2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:302]     http filter #2
[2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:303]       name: envoy.router
[2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:307]     config: {}
[2019-04-12 15:20:59.804][65][debug][config] [source/server/listener_manager_impl.cc:627] add warming listener: name=ambassador-listener-80, hash=1783155174245818883, address=0.0.0.0:80
[2019-04-12 15:20:59.804][65][debug][init] [source/common/init/manager_impl.cc:45] init manager Listener ambassador-listener-80 contains no targets
[2019-04-12 15:20:59.804][65][debug][init] [source/common/init/watcher_impl.cc:14] init manager Listener ambassador-listener-80 initialized, notifying ListenerImpl
[2019-04-12 15:20:59.805][65][debug][config] [source/server/listener_manager_impl.cc:627] warm complete. updating active listener: name=ambassador-listener-80, hash=1783155174245818883, address=0.0.0.0:80
[2019-04-12 15:20:59.805][65][debug][config] [source/server/listener_manager_impl.cc:627] draining listener: name=ambassador-listener-80, hash=5292162044335998753, address=0.0.0.0:80
[2019-04-12 15:20:59.805][65][info][upstream] [source/server/lds_api.cc:74] lds: add/update listener 'ambassador-listener-80'
[2019-04-12 15:20:59.806][65][debug][config] [source/common/config/grpc_mux_impl.cc:104] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration
[2019-04-12 15:20:59.806][65][debug][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC config for type.googleapis.com/envoy.api.v2.Listener accepted with 1 resources with version v40
[2019-04-12 15:20:59.806][65][debug][config] [source/common/config/grpc_mux_impl.cc:118] Received gRPC message for type.googleapis.com/envoy.api.v2.Cluster at version v40
[2019-04-12 15:20:59.806][65][debug][config] [source/common/config/grpc_mux_impl.cc:96] Pausing discovery requests for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
[2019-04-12 15:20:59.811][65][info][upstream] [source/common/upstream/cluster_manager_impl.cc:483] add/update cluster cluster_extauth_auth_service_svc_8080 starting warming
[2019-04-12 15:20:59.811][65][debug][config] [source/common/config/grpc_mux_impl.cc:96] Pausing discovery requests for type.googleapis.com/envoy.api.v2.Cluster
[2019-04-12 15:20:59.811][65][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
[2019-04-12 15:20:59.811][65][debug][upstream] [source/common/upstream/cds_api_impl.cc:110] cds: add/update cluster 'cluster_extauth_auth_service_svc_8080'
[2019-04-12 15:20:59.811][65][debug][config] [source/common/config/grpc_mux_impl.cc:104] Resuming discovery requests for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
[2019-04-12 15:20:59.811][65][debug][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC config for type.googleapis.com/envoy.api.v2.Cluster accepted with 3 resources with version v40
[2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/upstream_impl.cc:1358] DNS hosts have changed for auth-service-svc
[2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/upstream_impl.cc:721] initializing secondary cluster cluster_extauth_auth_service_svc_8080 completed
[2019-04-12 15:20:59.812][65][debug][init] [source/common/init/manager_impl.cc:45] init manager Cluster cluster_extauth_auth_service_svc_8080 contains no targets
[2019-04-12 15:20:59.812][65][debug][init] [source/common/init/watcher_impl.cc:14] init manager Cluster cluster_extauth_auth_service_svc_8080 initialized, notifying ClusterImplBase
[2019-04-12 15:20:59.812][65][info][upstream] [source/common/upstream/cluster_manager_impl.cc:496] warming cluster cluster_extauth_auth_service_svc_8080 complete
[2019-04-12 15:20:59.812][75][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:517] adding TLS cluster cluster_extauth_auth_service_svc_8080
[2019-04-12 15:20:59.812][76][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:517] adding TLS cluster cluster_extauth_auth_service_svc_8080
[2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:517] adding TLS cluster cluster_extauth_auth_service_svc_8080
[2019-04-12 15:20:59.812][75][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:978] membership update for TLS cluster cluster_extauth_auth_service_svc_8080 added 1 removed 0
[2019-04-12 15:20:59.812][76][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:978] membership update for TLS cluster cluster_extauth_auth_service_svc_8080 added 1 removed 0
[2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:978] membership update for TLS cluster cluster_extauth_auth_service_svc_8080 added 1 removed 0
[2019-04-12 15:20:59.812][65][debug][config] [source/common/config/grpc_mux_impl.cc:104] Resuming discovery requests for type.googleapis.com/envoy.api.v2.Cluster
....

Any hints as to why the path is not being detected and routed to the external auth service?

-- mithrandir
api-gateway
envoyproxy
kubernetes

Similar Questions

Specify Nginx ingress for both web and api
How to use large volumes of data in Kubeflow?
How to monitor a spring-boot service by creating a ServiceMonitor?
Where default kubernetes service gets a list of endpoints?
GKE: Ingress for internal load balance does not work
Kubernetes Multi Master setup failed
Kubernetes Executor does not spawn Kubernetes Pod Operator

1 Answer

4/12/2019

The path_prefix: "/api" does not create a mapping for your auth-service, it only tells that /api will be appended to your requests, and that the auth-service needs that prefix. When you define an auth service in Ambassador all your requests will be redirected to the auth service. If you need to make a request to the auth service directly, create a mapping for it.

Example: If you have a service with mapping:

apiVersion: ambassador/v1
kind:  Mapping
name:  myapp-mapping
prefix: /myapp/
service: myapp:8000

When you do curl $URL/myapp/ , the request will be redirected to auth-service at $URL/api/myapp. The auth-service needs to have an endpoint for /api/myapp. If it returns an HTTP status 200, then ambassador will send the original request to myapp. Otherwise, it will return the auth-service response to the client.
If you would like to use curl $URL/api/myapp then you could remove your prefix in your auth-service and change the prefix of myapp to /api/myapp/

Please read this tutorial for more details.

-- victortv
Source: StackOverflow