Istio DestinationRule gives upstream connect error or disconnect/reset before headers

11/1/2018

I try to get some basic routing between 2 apps deployed on a Google Cloud Kubernetes cluster with an lb ratio and I have this config:

apiVersion: v1
kind: Service
metadata:
  name: kubeapp
  labels:
    app: kubeapp
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: kubeapp
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kubeapp-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kubeapp
        version: kubeapp-v1
    spec:
      containers:
      - name: kubeapp-v1
        image: .......
        ports:
        - name: kubeapp-v1
          containerPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kubeapp-v2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kubeapp
        version: kubeapp-v2
    spec:
      containers:
      - name: kubeapp-v2
        image: .......
        ports:
        - name: kubeapp-v2
          containerPort: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeapp-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kubeapp
spec:
  hosts:
  - "*"
  gateways:
  - kubeapp-gateway
  http:
  - route:
    - destination:
        host: kubeapp
        port: 8080

which works perfectly and traffic goes 50/50 but when I try to add some basic rules for lb like:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kubeapp
spec:
  hosts:
  - "*"
  gateways:
  - kubeapp-gateway
  http:
  - route:
    - destination:
        host: kubeapp
        port:
          number: 8080
        subset: kubeapp-v1
      weight: 90
    - destination:
        host: kubeapp
        port:
          number: 8080
        subset: kubeapp-v2
      weight: 10
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: kubeapp
spec:
  host: kubeapp
  subsets:
  - name: kubeapp-v1
    labels:
      version: kubeapp-v1
  - name: kubeapp-v2
    labels:
      version: kubeapp-v2

I got upstream connect error or disconnect/reset before headers

I've tried to install Istio in all 3 modes and deploy it on different cluster nodes size (I saw that sometimes Istio has some bugs on some specific cluster size) and without success.

-- Padi
google-kubernetes-engine
istio
kubernetes

1 Answer

11/2/2018

A very common reason for this kind of problem is that your DestinationRule is causing an mTLS conflict. The issue is documented here.

-- Frank B
Source: StackOverflow