Currenly when updating a kubernetes secrets file, in order to apply the changes, I need to run kubectl apply -f my-secrets.yaml. If there was a running container, it would still be using the old secrets. In order to apply the new secrets on the running container, I currently run the command kubectl replace -f my-pod.yaml . I was wondering if this is the best way to update a running container secret, or am I missing something.
Thanks.
By design, Kubernetes won't push Secret updates to running Pods. If you want to update the Secret value for a Pod, you have to destroy and recreate the Pod. You can read more about it here.
The secret docs for users say this:
Mounted Secrets are updated automatically When a secret being already consumed in a volume is updated, projected keys are eventually updated as well. The update time depends on the kubelet syncing period.
Mounted secrets are updated. The question is when. In case a the content of a secret is updated does not mean that your application automatically consumes it. It is the job of your application to watch file changes in this scenario to act accordingly. Having this in mind you currently need to do a little bit more work. One way I have in mind right now would be to run a scheduled job in Kubernetes which talks to the Kubernetes API to initiate a new rollout of your deployment. That way you could theoretically achieve what you want to renew your secrets. It is somehow not elegant, but this is the only way I have in mind at the moment. I still need to check more on the Kubernetes concepts myself. So please bear with me.
Assuming we have running pod mypod [mounted secret as mysecret in pod spec]
We can delete the existing secret
kubectl delete secret mysecretrecreate the same secret with updated file
kubectl create secret mysecret <updated file/s>then do
kubectl apply -f ./mypod.yamlcheck the secrets inside mypod, it will be updated.
For k8s' versions >v1.15: kubectl rollout restart deployment $deploymentname: this will restart pods incrementally without causing downtime.
In case anyone (like me) want to force rolling update pods which are using those secrets. From this issue, the trick is to update an Env variable inside the container, then k8s will automatically rolling update entire pods
kubectl patch deployment mydeployment -p '{"spec":{"template":{"spec":{"containers":[{"name":"mycontainer","env":[{"name":"RESTART_","value":"'$(date +%s)'"}]}]}}}}'