K
Q

K8S with Traefik and HAP not getting real client IP on pods

4/7/2021

I have an external LB using HAProxy in order to have a HA k8s cluster. My cluster is in K3s from Rancher and it's using Traefik LB internally. I'm currently facing an issue where in my pods I'm getting the Traefik IP instead of the real client IP.

HAP Configuration:

# Ansible managed

defaults
  maxconn 1000
  mode http
  log global
  option dontlognull    
  log stdout local0 debug
  option httplog
  timeout http-request 5s
  timeout connect 5000
  timeout client 2000000
  timeout server 2000000

frontend k8s
  bind *:6443
  bind *:80
  bind *:443
  mode tcp
  option tcplog

  use_backend masters-k8s
    
backend masters-k8s
  mode tcp
  balance roundrobin

  server master01 master01.k8s.int.ntw
  server master02 master02.k8s.int.ntw
  
# end Ansible managed

Traefik Service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
  labels:
    app: traefik
    app.kubernetes.io/managed-by: Helm
    chart: traefik-1.81.0
    heritage: Helm
    release: traefik
spec:
  clusterIP: 10.43.250.142
  clusterIPs:
  - 10.43.250.142
  externalTrafficPolicy: Local
  ports:
  - name: http
    nodePort: 32232
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 30955
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: traefik
    release: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.0.1.1
    - ip: 10.0.1.11
    - ip: 10.0.1.12
    - ip: 10.0.1.2

With this configurations I never get my real IP on the pods, during some research I saw that people recommend to use send-proxy in the HAP like this:

  server master01 master01.k8s.int.ntw check send-proxy-v2
  server master02 master02.k8s.int.ntw check send-proxy-v2

But when I do so, all my cluster communication returns ERR_CONNECTION_CLOSED. If I'm looking at it correctly, this means its going from the HAP to the cluster and the cluster is somewhere rejecting the traffic.

Any clues what I'm missing here?

Thanks

-- Joaogl
haproxy
high-availability
k3s
kubernetes
traefik

Similar Questions

Kubernetes - Best way to create configMap and secret
Empty ADDRESS kubernetes ingress
Pubsub data affinity with Pod autoscalling
GKE DNS resolution errors
Securely shutdown a node from kubernetes cluster

1 Answer

4/15/2021

Well you have two options.

  1. use proxy protocol
  2. use X-Forwarded-For header

Option 1: proxy protocol

This option requires that both sites HAProxy and Traefik uses the proxy protocol, that's the reason why the people recommend "send-proxy-v2".
This requires also that all other clients which want to connect to Traefik MUST also use the proxy protocol, if the clients does not use the proxy protocol then you will get what you get a communication error.
As you configured HAProxy in TCP mode is this the only option to get the client ip to Traefik.

Option 2: X-Forwarded-For

I personally would use this option because it makes it possible to connect to traefik with any http(s) client.
This requires that you use the HTTP mode in haproxy and some more parameters like option forwardfor.

# Ansible managed

defaults
  maxconn 1000
  mode http
  log global
  option dontlognull    
  log stdout local0 debug
  option httplog
  timeout http-request 5s
  timeout connect 5s
  timeout client 200s
  timeout server 200s
  # send client ip in the x-forwarded-for header
  option forwardfor

frontend k8s
  bind *:6443 v4v6 alpn h2,http/1.1 ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/
  bind *:80   v4v6 alpn h2,http/1.1 ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/
  bind *:443  v4v6 alpn h2,http/1.1 ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/

  use_backend masters-k8s

backend masters-k8s

  balance roundrobin

  server master01 master01.k8s.int.ntw check
  server master02 master02.k8s.int.ntw check

# end Ansible managed

In the File "/etc/haproxy/letsencryptauthorityx3.pem" are the CA's for the backends, in the directory "/etc/ssl/haproxy/" are the certificates for the frontends.
Please take a look into the documentation about the crt keyword.
You have also to configure traefik to allow the header from haproxy forwarded-headers

-- Aleksandar
Source: StackOverflow