K
Q

x-b3-sampled header is always set to 0 when accessing service through ingress controller

8/20/2020

I have Kubernetes 1.17.5 and Istio 1.6.8 installed with demo profile.

And here is my test setup nginx-ingress-controller -> proxy<->ServiceA -> proxy<->ServiceB

  • Proxies for serviceA and serviceB are auto-injected by Istio (istio-injection=enabled)
  • Nginx ingress controller does not have tracing enabled and has no envoy proxy as a sidecar
  • ServiceA passes tracing headers down to ServiceB
  • I'm trying to trace calls from ServiceA to ServiceB and do not care about Ingress->ServiceA span at the moment

When I'm sending requests to ingress controller I can see that ServiceA receives all required tracing headers from the proxy

x-b3-traceid: d9bab9b4cdc8d0a7772e27bb7d15332f
x-request-id: 60e82827a270070cfbda38c6f30f478a
x-envoy-internal: true
x-b3-spanid: 772e27bb7d15332f
x-b3-sampled: 0
x-forwarded-proto: http

Problem is x-b3-sampled is always set to 0 and no spans/traces are getting pushed to Jaeger

Few things I've tried 1. I've added Gateway and VirtualService to ServiceA to expose it through Istio ingressgateway. When I send traffic through ingressgateway everything works as expected. I can see traces ingress-gateway->ServiceA->ServiceB in the JaegerUI 2. I've also tried to install Istio with custom config and play with tracing related parameters with no luck.

Here is the config I've tried to use

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        sampling: 100
  addonComponents:
    tracing:
      enabled: true
    grafana:
      enabled: false
    istiocoredns:
      enabled: false
    kiali:
      enabled: false
    prometheus:
      enabled: false
  values:
    tracing:
      enabled: true
    pilot:
      traceSampling: 100
-- arkadi4
istio
kubernetes
trace

Similar Questions

Loadbalancer IP/External IP is not getting generated on GKE cluster after using traefik Ingress controller
Kubernetes autoscaling not fast enough
Unauthorized when trying to allow nodes to join a Kubernetes cluster
Trying to create SQL instance in Google Cloud - Kubernetes
Is "replace" in kubernetes equivalent to 'update'?

1 Answer

8/25/2020

After few days of digging I've figured it out. Problem is in the format of the x-request-id header that nginx ingress controller uses.

Envoy proxy expects it to be an UUID (e.g. x-request-id: 3e21578f-cd04-9246-aa50-67188d790051) but ingrex controller passes it as a non-formatted random string (x-request-id: 60e82827a270070cfbda38c6f30f478a). When I pass properly formatted x-request-id header in the request to ingress controller its getting passed down to envoy proxy and request is getting sampled as expected. I also tried to remove x-request-id header from the request from ingress controller to ServiceA with a simple EnvoyFilter. And it also works as expected. Envoy proxy generates a new x-request-id and request is getting traced.

-- arkadi4
Source: StackOverflow