I need to scale my application so that it won't get banned for passing request rate-limit of a site it uses frequently (which allow up to X requests per minute per IP).
I meant to use kubernetes and split the requests between multiple workers, but I saw that all the pods get the same external IP. so what can I do?
do not worry a bout getting one external IP because if you have 3 worker and one master like below
worker1 192.168.1.10
worker2 192.168.1.11
worker3 192.168.1.12
master 192.168.1.13and forexample if you expose nginx on 30000 port the kubernetes open this port in every nod and you can access it by
curl 192.168.1.10:30000
curl 192.168.1.11:30000
curl 192.168.1.12:30000
curl 192.168.1.13:30000and if you want to every worker have one pod you can use DaemonSet or you can use label to the node that you want
This probably has less to do with your Kubernetes implementation and more to do with your network setup. It would depend on the source of the "exernal IP" you're referencing: is it given to you by your ISP? If you google "what is my ip", does it match the single IP you're talking about? If so, then you would need to negotiate with your ISP for additional external IPs.
Worth Noting that @JamesJJ is correct. Using additional IPs to 'trick' the API into allowing more connections is most likely a violation of that site's TOS and may result in your access getting terminated.
If you run in cloud you can create worker nodes with Public IP addresses. Then your pods will use node's public IP address. And then you can somehow distribute your pods across nodes using multiple replicas or DaemonSet.
I used kubernetes DaemonSet to attach pod to each node, and instead of scaling by changing deployment, I'm scaling by adding new nodes.