We are pulling images from a private ECR (AWS) running Kubernetes that ships with Docker for Mac (minikube). We have a secret called aws-cred. I created it using:

kubectl create secret docker-registry aws-creds --docker-server=OUR-ACCOUNT.ecr.eu-central-1.amazonaws.com --docker-username=AWS --docker-password=SUPER_LONG_TOKEN --docker-email=foo@bar.com

together with this in my deployments:

"imagePullSecrets":[{"name":"aws-creds"}]

SUPER_LONG_TOKEN I get from running:

aws ecr get-login --region eu-central-1 --profile default --no-include-email

Of course the token expires after a few hours and I tried to refresh the secret. First I deleted the secret:

kubectl delete secret aws-creds

Then basically repeated the steps above, fetching a fresh token. However I noticed, that I still cannot pull from our ECR getting a AWS ECR: no basic auth credentials error in minikube.

When I repeat the process, but I rename the secret, i.e. to aws-creds-2, everything works. I suspect there is some kind of caching in place. Indeed I verified this by using:

kubectl get secret aws-cred --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode

and I can see that the password value stays the same, even after deleting and re-creating the secret. This is a bit unintuitive to me, how should I update my secret instead?