I need to exclude specific host from the EnvoyFilter that looks like this:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authn-filter
spec:
  workloadLabels:
    istio: ingressgateway
  filters:
  - filterConfig:
      httpService:
        serverUri:
          uri: http://authservice.$(namespace).svc.cluster.local
          cluster: outbound|8080||authservice.$(namespace).svc.cluster.local
          failureModeAllow: false
          timeout: 10s
        authorizationRequest:
          allowedHeaders:
            patterns:
            - exact: "cookie"
            - exact: "X-Auth-Token"
        authorizationResponse:
          allowedUpstreamHeaders:
            patterns:
            - exact: "kubeflow-userid"
      statusOnError:
        code: GatewayTimeout
    filterName: envoy.ext_authz
    filterType: HTTP
    insertPosition:
      index: FIRST
    listenerMatch:
      listenerType: GATEWAY

The problem is that the filter applies to the default istio ingress gateway which affects all traffic that is coming through that gateway, i would like to have some hosts that could be excluded / whitelisted from the filter.