Unable to access websocket over Kubernetes ingress

8/11/2018

I have deployed two services to a Kubernetes Cluster on GCP:

One is a Spring Cloud Api Gateway implementation:

apiVersion: v1
kind: Service
metadata:
  name: api-gateway
spec:
  ports:
  - name: main
    port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: api-gateway
    tier: web
  type: NodePort

The other one is a backend chat service implementation which exposes a WebSocket at /ws/ path.

apiVersion: v1
kind: Service
metadata:
 name: chat-api
spec:
  ports:
  - name: main
    port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: chat
    tier: web
  type: NodePort

The API Gateway is exposed to internet through a Contour Ingress Controller:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: api-gateway-ingress
  annotations:
    kubernetes.io/tls-acme: "true"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
    ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
  - secretName: api-gateway-tls
    hosts:
    - api.mydomain.com.br
  rules:
  - host: api.mydomain.com.br
    http:
      paths:
      - backend:
          serviceName: api-gateway
          servicePort: 80

The gateway routes incoming calls to /chat/ path to the chat service on /ws/:

@Bean
public RouteLocator routes(RouteLocatorBuilder builder) {
    return builder.routes()
            .route(r -> r.path("/chat/**")
                    .filters(f -> f.rewritePath("/chat/(?<segment>.*)", "/ws/(?<segment>.*)"))
                    .uri("ws://chat-api"))
            .build();
}

When I try to connect to the WebSocket through the gateway I get a 403 error:

error: Unexpected server response: 403

I even tried to connect using http, https, ws and wss but the error remains.

Anyone has a clue?

-- Marcos J.C Kichel
kubernetes
kubernetes-ingress
spring
websocket

1 Answer

9/14/2018

I had the same issue using Ingress resource with Contour 0.5.0 but I managed to solve it by upgrading Contour to v0.6.0-beta.3 with IngressRoute (be aware, though, that it's a beta version).

You can add an IngressRoute resource (crd) like this (remove your previous ingress resource):

#ingressroute.yaml
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: api-gateway-ingress
  namespace: default
spec:
  virtualhost:
    fqdn: api.mydomain.com.br
    tls:
      secretName: api-gateway-tls
  routes:
    - match: /
      services:
        - name: api-gateway
          port: 80
    - match: /chat
      enableWebsockets: true # Setting this to true enables websocket for all paths that match /chat
      services:
        - name: api-gateway
          port: 80

Then apply it

Websockets will be authorized only on the /chat path.

See here for more detail about Contour IngressRoute.

-- Khaly
Source: StackOverflow