K
Q

Question

GKE Ingress dropping websocket connections when using https

8/1/2018

I have an Ingress (the default GKE one) which is handling all the SSL before my services. One of my services is a WebSocket service (python autobahn). When I am exposing the service using LoadBalancer and not passing throw the ingress, using ws:// everything us working good. When instead I am exposing it using NodePort and passing through the ingress I am constantly seeing connections that are dropping even when no client is connecting. Here are the autobahnlogs:

WARNING:autobahn.asyncio.websocket.WebSocketServerProtocol:dropping connection to peer tcp:10.156.0.58:36868 with abort=False: None

When I connect using a client with wss:// the connection is successful but a disconnection happens every few seconds (could not get a consistent number). Although I do not think it is related I changed the timeout of the related backend service in GCE to 3600 sec and also tried to give it session affinity using both clientIP and cookie but none seems to stop the dropping connections. Here is my ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ .Values.ingressName }}-ingress
  annotations:
      kubernetes.io/ingress.global-static-ip-name: {{ .Values.staticIpName }}-static-ip
  labels:
    oriient-app: "rest-api"
    oriient-system: "IPS"
spec:
  tls:
  - secretName: sslcerts
  rules:
  - host: {{ .Values.restApiHost }}
    http:
      paths:
      - backend:
          serviceName: rest-api-internal-service
          servicePort: 80
  - host: {{ .Values.dashboardHost }}
    http:
      paths:
      - backend:
          serviceName: dashboard-internal-service
          servicePort: 80
  - host: {{ .Values.monitorHost }}
    http:
      paths:
      - backend:
          serviceName: monitor-internal-service
          servicePort: 80
  - host: {{ .Values.ipsHost }}
    http:
      paths:
      - backend:
          serviceName: server-internal-ws-service
          servicePort: 80

The ws service is the "server-internal-ws-service". Any suggestions?

-- Yehuda

google-compute-engine

google-kubernetes-engine

kubernetes-ingress

websocket

1 Answer

8/1/2018

I did not solve the issue, but I did walk around it by exposing my wss with a LoadBalancer service and I implemented the secure layer of WebSocket by myself. I saved the certificate (the private key and full chain public key - pem format) as a secret and mounted it as volume and then in the python used the SSLContex to and passed it to the asyncio loop create server.

For creating the certificate secret create a yaml:

apiVersion: v1
kind: Secret
type: tls
metadata:
  name: sslcerts
data:
  # this is base64 of your pem fullchain and private key
  tls.crt: XXX 
  tls.key: YYY

and then

kubectl apply -f [path to the yaml above]

In your server deployment mount the secret:

    apiVersion: apps/v1beta2
    kind: Deployment
    metadata:
      labels:
        ...
      name: server
    spec:
      replicas: {{ .Values.replicas }}
      selector:
        matchLabels:
          ...
      template:
        metadata:
          labels:
            ...
        spec:
          volumes:
          - name: wss-ssl-certificate
            secret:
              secretName: sslcerts
       containers:
        - image: ...
        imagePullPolicy: Always
        name: server
        volumeMounts:
          - name: wss-ssl-certificate
            mountPath: /etc/wss

And in the python code:

 sslcontext = ssl.SSLContext()
 sslcontext.load_cert_chain(/etc/wss/tls.crt, /etc/wss/tls.key)
 wssIpsClientsFactory = WebSocketServerFactory()
 ...        
 loop = asyncio.get_event_loop()
 coro = loop.create_server(wssIpsClientsFactory, '0.0.0.0', 9000, ssl=sslcontext)
 server = loop.run_until_complete(coro)

Hope it helps someone

-- Yehuda

Source: StackOverflow

KQ

GKE Ingress dropping websocket connections when using https

Similar Questions

1 Answer

K
Q