We have some weird memory leaking issues with our containers where the longer they live, the more resources they take. We do not have the resources at the moment to look into these issues (as they don't become problems for over a month) but would like to avoid manual work to "clean up" the bloated containers.
What I'd like to do is configure our deployments in such a way that "time alive" is a parameter for the state of a pod, and if it exceed a value (say a couple days) the pod is killed off and a new one is created. I'd prefer to do this entirely within kubernetes, as while we will eventually be adding a "health check" endpoint to our services, that will not be able to be done for a while.
What is the best way to implement this sort of a "max age" parameter on the healthiness of a pod? Alternatively, I guess we could trigger based off of resource usage, but it's not an issue if the use is temporary, only if the resources aren't released after a short while.
The easiest way is to put a hard resource limit on memory that is above what you would see in a temporary spike: at a level that you'd expect to see over say a couple of weeks.
It's probably a good idea to do this anyhow, as k8s will schedule workloads based on requested resources, not their limit, so you could end up with memory pressure in a node as the memory usage increases.
One problem is would if you have significant memory spikes, the pod restart where k8s kills your pod would probably happen in the middle of some workload, so you'd need to be able to absorb that effect.
So, from the documentation it would look something like this (and clearly Deployment would be preferable to a raw Pod as shown below, this example can be carried over into a PodTemplateSpec):
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: ccccc
image: theimage
resources:
requests:
memory: "64Mi"
limits:
memory: "128Mi"