After the Pod is injected into the sidecar of istio, the websocket connection will be interrupted abnormally

12/2/2019

The backend uses Springboot to provide a WebSocket connection and sets the maximum idle time to 3 minutes. The program runs well in local. After 3 minutes of idle, the connection will be disconnected as scheduled. It can also be accessed normally by node port service when deployed in Kubernetes.

But when I inject sidecar into this backend pod, there is a problem. The connection doesn't work properly, often breaks, and is completely irregular. Sometimes when the frontend and backend are sending messages, it is interrupted suddenly. Sometimes it is interrupted after about 2 minutes idle. And sometimes the connection can only last for tens of seconds.

When the connection is interrupted, the backend will throw java.io.EOFException, and the frontend will receive the on close event.

This phenomenon will occur as long as a sidecar is injected into the pod(Even if I use node port service to access the pod). Also, I did a test, I used Nginx to transfer the request to port 31380 of istio-ingressgateway, and configured the gateway vs and dr as follows. But the result is the same.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: msapi
    version: product
  name: msapi
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      run: msapi
  template:
    metadata:
      labels:
        run: msapi
    spec:            
      containers:
      - env:
        - name: JAVA_OPTS
          valueFrom:
            configMapKeyRef:
              key: jvm.options
              name: test-config
        image: test/msapi:1.0.0
        imagePullPolicy: Always
        name: msapi
        ports:
        - containerPort: 9000
          protocol: TCP   
---
apiVersion: v1
kind: Service
metadata:
  name: msapi
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 9000
  selector:
    run: msapi
  type: ClusterIP
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-test
  namespace: test
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*.ingress.xxx.com'
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: msapi
  namespace: test
spec:
  gateways:
  - ingress-test
  hosts:
  - msapi.ingress.xxx.com
  http:
  - match:
    - headers:
        cookie:
          regex: ^(.*?; ?)?(version=pre)(;.*)?$
    route:
    - destination:
        host: msapi
        subset: pre
  - route:
    - destination:
        host: msapi
        subset: product
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: msapi
  namespace: test
spec:
  host: msapi
  subsets:
  - labels:
      version: product
    name: product
  - labels:
      version: pre
    name: pre
-- Li Yongsheng
istio
kubernetes
websocket

1 Answer

12/4/2019

The problem here was websocketUpgrade, one line but important one.

As I could find on github there

Support for websockets is enabled by default in Istio from version 1.0: https://godoc.org/istio.io/api/networking/v1alpha3#HTTPRoute

And OP provided another one there

websocketUpgrade was removed some time ago, and is no longer needed.

So it should work without adding it to virtual service.

HOWEVER

As showed on github issue and confirmed by OP You still have to add it.

I found that only need to add conf of "websocketUpgrade: true".

So if you have same issue, you should try add weboscketUpgrade to your virtual service yaml.

If that doesn't work, there is another idea on github how to fix this.

-- jt97
Source: StackOverflow