I'm trying to develop a scraping app in which I run a lot of different selenium scripts that are all dependent upon each other. I figured that using Kubernetes would be a good idea for this. In order for this to work, I need the scripts to be able to communicate with each other so I can trigger them to run after each other. How can I accomplish this?
This is an example of some logic that I'm want to perform:
I would appreciate any help that I can get!
The concept of Jobs sounds exactly like the stuff you'd like to achieve.
A Job creates one or more Pods and ensures that a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up the Pods it created.
A simple case is to create one Job object in order to reliably run one Pod to completion. The Job object will start a new Pod if the first Pod fails or is deleted (for example due to a node hardware failure or a node reboot).
You can also use a Job to run multiple Pods in parallel.
Additionally, you might be interested in a concept of Init Containers.
Init containers are exactly like regular containers, except:
Init containers always run to completion. Each init container must complete successfully before the next one starts..
If a Pod’s init container fails, Kubernetes repeatedly restarts the Pod until the init container succeeds.
And PV+PVC to store and share the data (JSON,etc)