I have a micro-service deployed on k8s which talks to an external cloud service. The external service has api rate limits on how many calls each apptoken can make. We can create multiple apptokens upfront.
We typically run 2-5 pods and would like to dedicate a token each pod so we avoid overlaps and exceeding the rate limits.
What is the simplest way to assign a token to each pod at start time, given the deployment setup is exactly the same for all and pods keep recycle dynamically?
The Right Way probably involves some sort of dedicated service to hand out these credentials. Potentially it could use the Kubernetes API to watch pods to see if they exited, or require services to periodically send a heartbeat to keep their current token. A shared Redis could be a lightweight way to maintain state without really having a server. If you were feeling fancy, you could write a Kubernetes controller that saw pods created with some label, and created a matching Secret object that had the secret.
If you're really trying to avoid this, you can run your application in a StatefulSet. This will have the property that pod names are assigned sequentially, myservice-0
, myservice-1
, ..., and so each pod knows which one it is. Then you can create a Secret with the tokens where the keys of the Secret are the expected pod names. The service would need code to retrieve the values from a mounted config file, or an init container would need to copy the correct value into a known place. The replicated MySQL example in the Kubernetes documentation talks more about this init-container setup. If your pods are stopping frequently or if you need to scale the setup up and down frequently, you could hit some operational trouble because of the constraints around StatefulSet utilization (if you have 5 replicas, you can't just stop #2 and leave it off; you must stop the highest-numbered one).
Another hacky approach would be to stand up a Redis container that pretty much just contained a counter. Write a small script in your choice of language that mounts the Secret with the tokens as a file, increments the counter (maybe wrapping around), and writes the n'th token into a file. Then you can have a Deployment running that as an init script. Here you can easily have two pods running with the same token, but if they restart routinely it won't be for too long.