K
Q

TLS handshake fails intermittently when using HAProxy Ingress Controller

9/29/2020

I'm using HAProxy Ingress Controller (https://github.com/helm/charts/tree/master/incubator/haproxy-ingress) for TLS-termination for my app. I have a simple Node.JS server listening on 8080 for HTTP, and 1935 as a simple echo server (not HTTP).
And I use HAProxy Ingress controller to wrap the ports in TLS. (8080 -> 443 (HTTPS), 1935 -> 1936 (TCP + TLS))
I installed HAProxy Ingress Controller with

helm upgrade --install haproxy-ingress incubator/haproxy-ingress \
   --namespace test \
  -f ./haproxy-ingress-values.yaml \
  --version v0.0.27

, where the content of haproxy-ingress-values.yaml is 

controller:
  ingressClass: haproxy
  replicaCount: 1
  service:
    type: LoadBalancer
  tcp:
    1936: "test/simple-server:1935:::test/ingress-cert"
  nodeSelector:
    "kubernetes.io/os": linux
defaultBackend:
  nodeSelector:
    "kubernetes.io/os": linux

And here's my ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "haproxy"
spec:
  tls:
  - hosts:
    secretName: ingress-cert
  rules:
    - http:
        paths:
        - path: /
          backend:
            serviceName: "simple-server"
            servicePort: 8080

The cert is self-signed. If I test the TLS handshake with 

echo | openssl s_client -connect "<IP>":1936

Sometimes (about 1/3 of the times) it fails with 

CONNECTED(00000005)
139828847829440:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:332:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 316 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

The same problem doesn't happen for 443 port.
See here for the details of the settings to reproduce the problem.

edit
As pointed out by @JoaoMorais, it's because the default statistic port is 1936. Although I didn't turn on statistics, it seems like it still interferes with the behavior.
There're two solutions that work for me.

  • Change my service's 1936 port to another
  • Change the stats port by adding values like below when installing the haproxy-ingress chart.
controller:
  stats:
    port: 5000
-- someone
haproxy-ingress
kubernetes
ssl

Similar Questions

OpenShift - External Database Environment Variables
kubectl replace -f says it has replaces a resource, but it actually hasn't
Deploy Nodejs app on google kubernetes engine using Docker and PM2
How to create and mount common data for all pods
How to broadcast gRPC streams inter pods

1 Answer

9/29/2020

HAProxy by default allows to reuse the same port number across the same or other frontend/listen sections and also across other haproxy process. This can be changed adding noreuseport in the global section.

The default HAProxy Ingress configuration uses port number 1936 to expose stats. If such port number is reused by eg a tcp proxy, the incoming requests will be distributed between both frontends - sometimes your service will be called, sometimes the stats page. Changing the tcp proxy or the stats page (doc here) to another port should solve the issue.

-- Joao Morais
Source: StackOverflow