Golang High GC pause times on docker/kubernetes

3/8/2017

I am migrationg a web application written in Go from AWS Elastic Beanstalk to Kubernets and I noticed that the garbage collector pause times (I am using Newrelic to monitor the application) increased about a 100 when running the application.

I believe it is related with the CPU limiting that the Kubernet does.

Does anyone have any idea about what is really causing it? Is it possible to overcome it?

Below there is a small example of this difference.

Elastc Beanstalk:

GC Pause time - Elastic Beanstalk

Kubernets:

GC Pause time - Kubernets

After some tests and more research I discovered some interesting things. The CPU limit on Docker seems to have a great influence on GC time/pauses. After some tests I got the CPU limit to 500m which means about 1/2 CPU of a 8-core machine.

I set GOMAXPROCS = 1 and GOGC = 1000 and this lead to less and faster GC pauses, however the average memory usage increased.

Here are a 27h overview of Kubernets and Elastic Beanstalk

Kubernetes:

GC overview - Kubernetes

Elastic Beanstalk:

GC overview - Elastic Beanstalk

-- Tiago Queiroz
docker
go
kubernetes

0 Answers