Socket.io Ingress controller on Kubernetes (Can't estabilish a connection)

8/2/2021

I have a simple Node.js chat example which uses socket.io. I have tested this deployment with a docker container locally, and a web client succesfully connects to the websocket (socket.io). The problem arise when I try to deploy this on a Kubernetes cluster with an ingress Controller.

My question is, is there anything special socket.io specifically needs when using an ingress controller?

The error message I get is <<browser name>> Can't estabilish a connection to the server at ws://<<address>>/socket.io/EIO=3&transport=websocket

Below is the simple implementation of the server side

var io = socket(server);
io.on('connection', (socket)=>{
    console.log('made socket connection')
    socket.on('chat', (message)=>{
        message:message.value
    })

Below is the client:

var socket = io.connect("http://localhost:3002", {
    upgrade: false,
    transports: ['websocket'],
    secure: True
});

My ingress yaml file looks like this

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ws-ingress
  namespace: websocket-ns
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/websocket-services: "ws-service"
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
    nginx.ingress.kubernetes.io/server-snippet: |
      http {
        server {
            listen 3002;
          location = / {
              proxy_set_header Upgrade "websocket";
              proxy_http_version 1.1;
              proxy_set_header X-Forwarded-Host $http_host;
              proxy_set_header X-Forwarded-Proto $scheme;
              proxy_set_header X-Forwarded-For $remote_addr;
              proxy_set_header Host $host;
              proxy_set_header Connection "upgrade";
              proxy_cache_bypass $http_upgrade;
            }
          upstream nodes {
              hash $remote_addr consistent;
              server app01:3002;
            }
      }
spec:
  rules:
    - host: "$host_address"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ws-service
                port:
                  number: 80

Service yaml:

apiVersion: v1
kind: Service
metadata:
  name: ws-service
  namespace: websocket-ns
  labels:
    app: ws-service
spec:
  type: LoadBalancer
  selector:
    app: ws-app
  ports:
    - port: 80
      protocol: TCP
      targetPort: 3002

Server deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws-app
  namespace: websocket-ns
  labels:
    app: ws-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ws-app
  template:
    metadata:
      labels:
        app: ws-app
    spec:
      containers:
        - name: ws-app
          image: themuchy/socketexample
          ports:
            - containerPort: 3002
              name: ws-app
-- Tinashe Muchabaiwa
kubernetes
kubernetes-ingress
nginx-ingress
socket.io
websocket

1 Answer

8/2/2021

This one is working with one

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: core-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/websocket-services: ws-service
    nginx.org/websocket-services: ws-service
  name: core-ingress
spec:
  rules:
  - host: test.io
    http:
      paths:
      - backend:
          serviceName: ws-service
          servicePort: 80
  tls:
  - hosts:
    - test.io
    secretName: core-prod

Nginx itself will upgrade the HTTP request and convert it to WebSocket. You don't have to add any annotation of the same.

Deployment and service, YAML configuration looking good however try removing the annotation server-snippet. This way traffic flow will be something like

LB > ingress (Connection upgrade) > service > pods
-- Harsh Manvar
Source: StackOverflow