K
Q

kubernetes ingress websockets (socket.io)

11/26/2019

I keep getting bad requests (400) when trying to open a websocket, and I'm trying to figure out why. The back-end is built on flask with flask-socketio. Here is my Docker container for the back-end:

FROM alpine:edge

RUN apk update
RUN apk add python3 py3-cffi py3-bcrypt libc-dev py3-psycopg2 py3-gevent
RUN pip3 install --upgrade pip
RUN pip3 install flask flask-restful flask-jwt-extended gunicorn requests flask-sqlalchemy flask-socketio

ADD ./rest-api /root/rest-api
ADD ./ui/dist/ui /root/ui

CMD ["gunicorn", "-k", "gevent", "-w", "1", "--bind", "0.0.0.0:3001", "--access-logfile", "-", "--chdir", "/root/rest-api/", "app:app"]

Here are my yaml files:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-global-ip
    networking.gke.io/managed-certificates: my-certificate
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/websocket-services: "my-service-web"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
spec:
  rules:
  - http:
      paths:
      - path: /grafana/*
        backend:
          serviceName: my-service-web
          servicePort: grafana-port
      - path: /*
        backend:
          serviceName: my-service-web
          servicePort: web-app-port
apiVersion: v1
kind: Service
metadata:
  name: my-service-web

spec:
  ports:
    - port: 3000
      name: grafana-port
      targetPort: grafana-port
      protocol: TCP
    - port: 3001
      name: web-app-port
      targetPort: web-app-port
      protocol: TCP
  selector:
    app: my-cloud
  type: NodePort

I can see from the logs that the requests reach the back-end container:

10.166.0.42 - - [26/Nov/2019:08:52:22 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.4.2.1 - - [26/Nov/2019:08:52:23 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.4.2.1 - - [26/Nov/2019:08:52:25 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.4.2.1 - - [26/Nov/2019:08:52:28 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.4.2.1 - - [26/Nov/2019:08:52:34 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.166.0.41 - - [26/Nov/2019:08:52:39 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.166.0.41 - - [26/Nov/2019:08:52:44 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.166.0.42 - - [26/Nov/2019:08:52:49 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"
10.4.2.1 - - [26/Nov/2019:08:52:54 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"

What I can't figure out is how to configure kubernetes ingress to follow flask-socketio configuration for nginx:

    location /socket.io {
        include proxy_params;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass http://127.0.0.1:5000/socket.io;
    }

How do I do the connection upgrade to websocket in kubernetes ingress?

Update: I instantiated SocketIO with logs enabled and got the following:

6f3a03945f174b039b033e887079b97d: Sending packet OPEN data {'sid': '6f3a03945f174b039b033e887079b97d', 'upgrades': [], 'pingTimeout': 60000, 'pingInterval': 25000}
6f3a03945f174b039b033e887079b97d: Sending packet MESSAGE data 0
6f3a03945f174b039b033e887079b97d: Received request to upgrade to websocket
10.166.0.43 - - [26/Nov/2019:10:19:45 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 400 11 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0
-- remus
flask
flask-socketio
kubernetes
kubernetes-ingress

Similar Questions

How to determine kubernetes pod ephemeral storage request and limit?
How to create a MySQL cluster within an Istio injected namespace?
How to calculate Percentage utilization of particular pod in kubernetes with metrics server?
Is there a way to run react Application on SpringBoot and to in Kubernetes?
Where does Kubernetes store cached images for use of "IfNotPresent"-CachePolicy and how to delete the cache?

1 Answer

12/9/2019

I almost had it right from the beginning. I changed the CMD in my Dockerfile to (as per Flask-socketio docs):

gunicorn -k geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 module:app

Here's the complete Dockerfile:

FROM alpine:edge

RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk update && apk upgrade
RUN apk add python3 py3-cffi py3-bcrypt libc-dev py3-psycopg2 py3-gevent-websocket
RUN pip3 install --upgrade pip
RUN pip3 install flask flask-restful flask-jwt-extended gunicorn requests flask-sqlalchemy flask-socketio

ADD ./rest-api /root/rest-api
ADD ./ui/dist/ui /root/ui

CMD ["gunicorn", "-k", "geventwebsocket.gunicorn.workers.GeventWebSocketWorker", "-w", "1", "--bind", "0.0.0.0:3001", "--timeout", "180", "--access-logfile", "-", "--chdir", "/root/rest-api/", "app:app"]
-- remus
Source: StackOverflow