To clarify, this is not a question about running Jenkins in Kubernetes, this is about deploying to Kubernetess from Jenkins.

I have recently settled on using Jenkins (and the workflow/pipeline plugin) to orchestrate our delivery process. Currently, I'm using the imperative style to deploy as per below:

stage 'Deploy to Integ'
  // Clean up old releases
  sh "kubectl delete svc,deployment ${serviceName} || true"
    def cmd = """kubectl run ${serviceName} --image=${dockerRegistry}/${serviceName}:${env.BUILD_NUMBER} --replicas=2 --port=${containerPort} --expose --service-overrides='{ "spec": { "type": "LoadBalancer"  }}' """
    // execute shell for the command above
  sh cmd

This works well because the ${env.BUILD_NUMBER} persists through the pipeline, making it easy for me to ensure the version I deploy is the same all the way through. The problem I have is that I would like to use the declarative approach as this isn't scalable, and I would like the definition in VCS.

Unfortunately, the declarative approach comes with the adverse effect of needing to explicitly state the version of the image (to be deployed) in the yaml. One way around this might be to use the latest tag, however this comes with its own risks. For example, lets take the scenario where I'm about to deploy latest to production and a new version gets tagged latest. The new latest may not have gone through testing.

I could get into changing the file programmatically, but that feels rather clunky, and doesn't help developers who have the file checked out to understand what is latest.

What have you done to solve this issue? Am I missing something obvious? What workflow are you using?