Istio 1.8 upstream connect error or disconnect/reset before headers. reset reason: connection failure

1/4/2021

I'm trying to learn how to use istio for canary deploy.

The idea would be to use a hybrid between Istio and Traefik ingress.

Unfortunately, I am having problems with the VirtualService, specifically, when I use the IP of the Traefik LoadBalancer, I get this error:

enter image description here

I'm testing it with a generic helloworld application. Below I copy the various settings:

Deployment helloworld-v1:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations: 
    env: prod
  labels:
    app: helloworld
    version: v1
  name: helloworld-v1
  namespace: university
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: helloworld
        version: v1
    spec:
      containers:
      - args:
        - -text=banana
        image: hashicorp/http-echo
        imagePullPolicy: IfNotPresent
        name: helloworld
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          limits:
            cpu: 50m
            memory: 128Mi

Deployment helloworld-v2: same as helloworld-v1 but with text=apple and different tag version

Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: helloworld
  name: helloworld
  namespace: university
spec:
  ports:
  - name: http-helloworld
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: helloworld
  sessionAffinity: None
  type: ClusterIP

VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: helloworld
  namespace: university
spec:
  hosts:
  - helloworld
  http:
  - route:
    - destination:
        host: helloworld
        subset: v1
      weight: 90
    - destination:
        host: helloworld
        subset: v2
      weight: 10

DestinationRule

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
  name: helloworld
  namespace: university
spec:
  host: helloworld
  subsets:
  - labels:
      version: v1
    name: v1
  - labels:
      version: v2
    name: v2

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  name: helloworld
  namespace: university
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: helloworld
          servicePort: http-helloworld
        path: /

I get some logs from pod and I obtain this:

2021-01-04T15:33:48.733757Z     warning envoy runtime   Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-01-04T15:33:48.771547Z     info    xdsproxy        Envoy ADS stream established
2021-01-04T15:33:48.771879Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2021-01-04T15:33:48.772087Z     warning envoy main      there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2021-01-04T15:33:48.867949Z     info    sds     resource:ROOTCA new connection
2021-01-04T15:33:48.868040Z     info    sds     Skipping waiting for gateway secret
2021-01-04T15:33:48.868347Z     info    sds     resource:default new connection
2021-01-04T15:33:48.868394Z     info    sds     Skipping waiting for gateway secret
2021-01-04T15:33:49.123541Z     info    cache   Root cert has changed, start rotating root cert for SDS clients
2021-01-04T15:33:49.123574Z     info    cache   GenerateSecret default
2021-01-04T15:33:49.123987Z     info    sds     resource:default pushed key/cert pair to proxy
2021-01-04T15:33:49.468337Z     info    cache   Loaded root cert from certificate ROOTCA
2021-01-04T15:33:49.468588Z     info    sds     resource:ROOTCA pushed root cert to proxy
2021-01-04T15:33:49.724483Z     warning envoy filter    mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2021-01-04T15:33:49.726036Z     warning envoy filter    mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2021-01-04T15:33:51.398713Z     info    Envoy proxy is ready
[2021-01-04T15:34:41.103Z] "GET /favicon.ico HTTP/1.1" 503 UF "-" 0 91 0 - "127.0.0.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "7bcaf3da-8fc3-958f-8907-a83d2dedf274" "10.128.252.134" "127.0.0.1:80" inbound|80|| - 10.28.252.77:80 127.0.0.1:0 
- default
**[2021-01-04T15:34:46.232Z] "GET /favicon.ico HTTP/1.1" 503 UF "-" 0 91 0 - "127.0.0.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "0e7f0815-d3d1-9b6c-82b3-ae171aafd8fd" "10.128.252.134" "127.0.0.1:80" inbound|80|| - 10.28.252.77:80 127.0.0.1:0 **
- default
[2021-01-04T15:38:04.023Z] "GET / HTTP/1.1" 503 UF "-" 0 91 0 - "127.0.0.1" "curl/7.65.1" "210c09e2-20e2-9a0e-9241-ec662fb260d0" "10.128.252.134" "127.0.0.1:80" inbound|80|| - 10.28.252.77:80 127.0.0.1:0 - default
[2021-01-04T15:38:10.014Z] "GET /favicon.ico HTTP/1.1" 503 UF "-" 0 91 0 - "127.0.0.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "b571e684-c15c-90b1-8d7d-d0dc52f9e71d" "10.128.252.134" "127.0.0.1:80" inbound|80|| - 10.28.252.77:80 127.0.0.1:0 
- default

Do you have any ideas?

Thanks for help

NB: The namespace is correctly injected

-- Martina Pesca Bellio
docker
istio
istio-kiali
kubernetes
kubernetes-ingress

0 Answers