We have python services running in pods in a kubernetes cluster. The services are setup to receive their log-level from an environment variable. Those env vars are set during the deployment of the service in a gitlab pipeline. For debugging purposes I want to be able to just change the env var on a single pod and restart it, without having to redeploy the service from gitlab.
Before we moved to kubernetes, we were running our containers in rancher, where the described change was very easy to do in the GUI. Change the env var -> hit update -> container restarts automatically.
I found this article that suggest to change the replica set using a command like
kubectl set env rs [REPLICASET_NAME] [ENV_VAR]=[VALUE]And then terminating the pod, after which it will be recreated with the env var set accordingly.
But it also states to
Never do it on a production system.
Never even do this on a dev environment without taking care in how it may impact your deployment workflow.
Is that the only / best way to achieve my goal of quickly changing an env var in a running pod for debug purposes?
Is that the only / best way to achieve my goal of quickly changing an env var in a running pod for debug purposes?
Short answer: Yes.
Long answer: I've never used or read up on Rancher, but I suspect that it was also changing the ReplicaSet or Deployment template env var, which triggered a Pod update. It's really the only way to change an env var in a Pod. You can't change the env vars on a running container or a running Pod. You can't do that in Docker containers, and you can't do it in Kubernetes, so I assume that you can't do it in Rancher. You can only restart a Pod with a different spec.
Why?
Because containers are just processes running on the host machine. Once the process is started, it's not possible to change a process's environment without resorting to nasty hacks.
If you're just concerned about the warnings that state to not do this in dev or prod, I would say that the same warnings apply to the Rancher workflow you described, so if you were willing to take the risks there, it won't be any different here.