I have a cluster to manage which has Pods running with Image-CachePolicy set to "IfNotPresent".
containers:
- name: test-app
image: myimages/test-app
imagePullPolicy: "IfNotPresent"I am already familiar with the difference between IfNotPresent and Always.
In our case, if the Pod gets respawned, K8s will due to "IfNotPresent" load the image from some cache instead of fetching the image from docker-hub.
What we want to achieve is to find this place where kubectl is caching the image and delete the cache in order to make K8s re-fetching the image on the next time, when the Pod gets restarted.
We don't want to apply a new Policy with imagePullPolicy set to Always because this would already fetch the latest version of the Pod, which we don't want yet. Instead, it shall only fetch new newest image on the next restart of the pod.
Is there any way to do this? Where to find this caching-location?
Kubernetes itself doesn't cache the images, it is the underlying OCI runtime that does the caching.
If the k8s cluster is using docker, an image can be removed from a node with:
docker image rm <name_or_checksum>Welcome to stackOverflow, related to where images are cached, as explained by @David Maze and @Matt above it depends on your OCI.
I would like to understand a little more about this statement
> We don't want to apply a new Policy with imagePullPolicy set to Always because this would already fetch the latest version of the Pod, which we don't want yet. Instead, it shall only fetch new newest image on the next restart of the pod.
When you mention "we don't want yet" does that mean your current image on that tag is not the right one yet. Is there a possibility to modify your CI/CD so that image tags (tag used in Kubernetes manifest) to be generated or created only based on a promotion flow (meaning only that at that time when you want that particular tag ready for Kubernetes clusters)? This is the normal way of handling images for Kubernetes uses.
The reason for my suggestion is, there could be several reasons why your pod can also get re-scheduled to other nodes (nodes without any cache and k8s can instruct OCI to refetch, as that node may not have this cache) at any point in time. Assuming pods lifetime might give some unexpected outcomes.
I would suggest stick to Kubernetes image policy and modify CI/CD process or promotion flows to handle which image to be ready for which Kubernetes environment.
If you still feel there is a need to modify and play with image caches, explain more on the actual use case, there could be other alternatives.
If you imagined Kubernetes was running Docker, it'd do the equivalent of:
docker images.docker pull it.docker run the resulting image.(imagePullPolicy: Always skips the first step and always pulls the image; Never skips the pull, and will fail if the image isn't already there.)
Each node will have its own copy of the images; where exactly depends on the specific container runtime in use. Each node in turn knows how to garbage collect images not currently in use, though old images will generally stick around until the local disk fills up (or until a tool like the cluster autoscaler deletes the node).
A general assumption in Kubernetes is that a given registry.example.com/name/image:tag string uniquely identifies an image. Your CI/CD system should cooperate with this, and assign a distinct tag to each image build (maybe based on the commit ID or a date stamp). If you do this, you shouldn't need to adjust imagePullPolicy: from its default or try to manually remove images from nodes.