In my k8s cluster I have two deployments: one for generating JWTs using a given key-pair and another one for validating JWTs using the same key-pair.
Both deployments using the same k8s secret which contains the key/pair.
When I want to revoke/update that key pair, how do I create consistency between the deployments? Practically, I want that all the generated JWTs will be validated although there are two different microservices, and not necessarily all the pods of both microservices will be updated at once for using the new keys.
How do I prevent such false alarms of validation failures?
In your Helm chart or Kustomize manifest, set a hash of the contents of the secret as an annotation on pod template, that will automatically trigger a re-deployment when the contents of the secret changes.
Alternatively your software could detect the changes in the files and reload them.
It is impossible to have a generic solution for this problem: you must coordinate all the parties yourself.
The common solution is to issue a new secret and let all the participants to accept both.
Then after some time to stop issuing the old version and remove it from everywhere.