We have an Image Service written in Golang.

1. It supports image operation like resize crop blur..
2. The RPS is around 400.
3. Pod Config : 16GB RAM and 8 cores
4. We deployed the application and observed for a day, it showed high core utilization
5. We introduced ballast(https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/) of 4GB and Sync pool(https://medium.com/a-journey-with-go/go-understand-the-design-of-sync-pool-2dde3024e277) to contain the core issues
6. Next we started observing high memory utilization.
7. Hence we reduced Ballast to 1GB, but still memory utilization is high
8. According to this article https://www.bwplotka.dev/2019/golang-memory-monitoring/ Goland version 1.12+ reported high RSS According to the article "This does not mean that they require more memory, it’s just optimization for cases where there is no other memory pressure."
9. To verify that we did a small POC on local machine to validate above and it worked.
10. Local Set up - Container memory - 500MB
11. The memory would continuously increase if it had and would remain there at 450MB until the pressure increases. As soon as the pressure increases the memory would go down to 4MB.
12. But this POC failed on Kubernetes cluster and the pods started crashing and restarting when the memory reached ~16 GB RAM on high RPS like 400.

Can someone suggest how can we contain this memory issue and why this POC failed on the cluster.

Let me know if more detail is required..