K
Q

How to update kubernetes deployment without updating image

5/11/2018

Background.

We are using k8s 1.7. We use deployment.yml to maintain/update k8s cluster state. In deployment.yml, pod's image is set to ${some_image}:latest. Once deployment is created, pod's image will update to ${some_image}:${build_num}, whenever there is code merge into master.

What happen now is, let's say if we need to modified the resource limited in deployment.yml and re-apply it. The image of deployment will be updated to ${some_image} :latest as well. We want to keep the image as it is in cluster state, without maintaining the actual tag in deployment.yml. We know that the replcas can be omitted in file, and it takes the value from cluster state by default.

Question,

On 1.7, the spec.template.spec.containers[0].image is required.

  1. Is it possible to apply deployment.yml without updating the image to ${some_image}:latest as well (an argument like --ignore-image-change, or a specific field in deployment.yml)? If so, how?

Also, I see the image is optional in 1.10 documentation.

  1. Is it true? if so, since which version?

--- Updates ---

CI build and deploy new image on every merge into master. At deploy, CI run the command kubectl set image deployment/app container=${some_image}:${build_num} where ${build_num} is the build number of the pipeline.

To apply deployment.yml, we run kubectl apply -f deployment.yml

-- benjah1
kubernetes

Similar Questions

Mixing local and remote Docker images repo?
Kubernates pass env variable to "kubectl create"
How to consume properties from configmaps in Java Spring boot application deployed through Helm
Best way to update Kubernetes secret from command line
Cannot access Longhorn UI even with LoadBalancer IP assigned [metalLB]
Not able to connect private github repository in apache ariflow helm chart in Kubernetes

4 Answers

5/14/2018

In case you don't want to deal with complex syntax in deployment.yaml in CI, you have the option to use a template processor. For example mustache. It would change the CI process a little bit:

  1. update image version in template config (env1.yaml)
  2. generate deployment.yaml from template deployment.mustache and env1.yaml
    $ mustache env1.yml deployment.mustache > deployment.yaml
  3. apply configuration to cluster.
    $ kubectl apply -f deployment.yaml

The main benefits:

  1. env1.yaml always contains the latest master build image, so you are creating the deployment object using correct image.
  2. env1.yaml is easy to update or generate at the CI step.
  3. deployment.mustache stays immutable, and you are sure that all that could possibly change in the final deployment.yaml is an image version.

There are many other template rendering solutions in case mustache doesn't fit well in your CI.

-- VAS
Source: StackOverflow
5/12/2018

Are you aware of the repo.example.com/some-tag@sha256:... syntax for pulling images from docker registry? It is almost exactly designed to solve the problem you are describing.

updated from a comment:

You're solving the wrong problem; the file is only used to load content into the cluster -- from that moment forward, the authoritative copy of the metadata is in the cluster. The kubectl patch command can be a surgical way of changing some content without resorting to sed (or worse), but one should not try and maintain cluster state outside the cluster

-- mdaniel
Source: StackOverflow
5/22/2018

Like Const above I highly recommend against using :latest in any docker image and instead use CI/CD to solve the version problem.

We have the same issue on the Jenkins X project where we have many git repositories and as we change things like libraries or base docker images we need to change lots of versions in pom.xml, package.json, Dockerfiles, helm charts etc.

We use a simple CLI tool called UpdateBot which automates the generation of Pull Requests on all downstream repositories. We tend to think of this as Continuous Delivery for libraries and base images ;). e.g. here's the current Pull Requests that UpdateBot has generated on the Jenkins X organisation repositories

Then here's how we update Dockerfiles / helm charts as we release, say, new base images: https://github.com/jenkins-x/builder-base/blob/master/jx/scripts/release.sh#L28-L29

-- James Strachan
Source: StackOverflow
5/12/2018

However, in deployment.yml file, we specified the latest tag of the image, because it is impossible to keep this field up-to-date

  • Using “:latest” tag is against best practices in Kubernetes deployments for a number of reasons - rollback and versioning being some of them. To properly resolve this you should maybe rethink you CI/CD pipeline approach. We use ci-pipeline or ci-job version to tag images for example.

Is it possible to update deployment without updating the image to the file specified. If so, how?

  • To update pod without changing the image you have some options, each with some constraints, and they all require some Ops gymnastics and introduce additional points of failure since it goes against recommended approach.
  • k8s can pull the image from your remote registry (you must keep track of hashes since your latest is out of your direct control - potential issues here). You can check used hash on local docker registry of a node that pod is running from.
  • k8s can pull the image from local node registry (you must ensure that on all potential nodes for running pods at “:latest” is on the same page in local registry for this to work - potential issues here). Once there, you can play with container’s imagePullPolicy such that when CI tool is deploying - it uses apply of yaml (in contrast to create) and sets image policu to Always, immediately folowing by apply of image policy of Never (also potential issue here), restricting pulling policy to already pulled image to local repository (as mentioned, potential issues here as well).
  • Here is an excerpt from documentation about this approach: By default, the kubelet will try to pull each image from the specified registry. However, if the imagePullPolicy property of the container is set to IfNotPresent or Never, then a local image is used (preferentially or exclusively, respectively). If you want to rely on pre-pulled images as a substitute for registry authentication, you must ensure all nodes in the cluster have the same pre-pulled images.
  • more about how k8s is handling images and why latest tagging can bite back is given here: https://kubernetes.io/docs/concepts/containers/images/
-- Const
Source: StackOverflow