HAProxy reverse ssl termination: Memory keeps growing. Memory leak?

2/14/2022

I have haproxy 2.5.1 in SSL termination config running in a container of a Kubernetes POD, the backend is an Scala App that runs in another container of same POD.

I have seen that I can put 500K connections in the setup and the RSS memory usage of HAProxy is 20GB. If I remove the traffic and wait 15 minutes the RSS memory drops to 15GB, but if I repeat the same exercise one or two more times, RSS for HAProxy will hit 30GB and HAProxy will be kill as I have a limit of 30GB in the POD for HAProxy.

The question here is if this behavior of continuous memory growth is expected?

Here is the incoming traffic: enter image description here

And here is the memory usage chart which shows how after 3 cycles of Placing Load and Removing Load, the RSS memory reached 30GB and then got killed (Just as an observation the two charts have different timezone but they belong to same execution)

enter image description here

-- Luis Serrano
haproxy
high-load
kubernetes
performance-testing
tcp

1 Answer

3/3/2022

We switched from Alpine based image(musl) into libc based image and that solved the problem. We got 5X increase on connection rate and memory growth gone too.

-- Luis Serrano
Source: StackOverflow