Istio circuit breaker not opening the circuit on consecutiveErrors when downstream service throws 5xx (500, 502, 503, 504) errors

4/16/2019

I have a couple of microservices, customer-service and customer-rating-service. The first one invokes the latter.

I have placed a circuit breaker on invocations to customer-rating-service, and forced this service to always throw a 5xx error to validate the circuit breaker. However customer-service always calls it, apparently the circuit is never open.

customer-rating-service - Istio Virtual Service

...
spec:
  hosts:
    - customer-rating-service
  gateways: ~
  http:
    - route:
        - destination:
            host: customer-rating-service
            subset: v1

customer-rating-service - Istio Destination Rule

...
spec:
  host: customer-rating-service
  trafficPolicy:
    outlierDetection:
      baseEjectionTime: 30s
      consecutiveErrors: 1
      maxEjectionPercent: 100
      minHealthPercent: 0
    tls:
      mode: ISTIO_MUTUAL
  subsets:
    - labels:
        version: v1
      name: v1

As you can see I have set consecutiveErrors: 1 so after the first invocation from customer-service to customer-rating-service, since this returns a 5xx error (I've tried throwing different errors 500, 502, 503...), the circuit should open. However every subsequent invocation reaches the customer-rating-service. Where is the problem?

NOTICE: there's only one instance of each service.

-- codependent
circuit-breaker
istio
kubernetes
openshift

1 Answer

9/4/2019

This feature works in Istio 1.3 (but not in 1.2). See the issue I raised at https://github.com/istio/api/issues/1068

-- Vinay B
Source: StackOverflow