I have a kubernetes deployment which uses secrets for environment variables. So the pod has env variables which are using the secret.
Now I updated my deployment manifest json and the secret json to remove some of the variables, but when I apply these files, I get CreateContainerConfigError for the pod, and in its description I see:
Couldn't find key FOO in Secret mynamespace/secret-mypod
I've deleted this key from my secret json and from the deployment manifest json. Why am I still getting this error?
I understand that I can delete the deployment and apply it to get it to work, but I don't want to do that.
What is the right way to handle this scenario?
I would do a replace on the deployment json. If the original deployment had the usual update settings, a new pod will be created with the new deployment config. If it starts ok, the old one will be deleted.
To be safer, you could create a new secret with some version indicator in the name, refer to the new secret name in the new deployment definition. That way if there's a problem with the new pod or the old needs to be redeployed, the old secret will still be there.