In the spirit to make the wording of my question in a clean and simplistic fashion, I try to avoid using word such as 'redundancy', 'distributed', 'clustering', 'orchestrating', 'fault tolerant', 'container'
If I am to write
, two very similar programs, each multi-threaded
My goal is to write A and B in such way that when A terminates unexpectedly, B will run instead. Since A and B share a centralized database, integrity of data is not the main concern here.
Question 1: what is the simplest and most elegant approach to enable the 'monitoring' mechanism needed to detect the termination of A and 'wake up' B, at each of the following scalability levels?
Level 1: one instance of each of A and B running on the same processor and RAM (e.g. should I use JMXConnector?)
Level 2: one instance of each of A and B running on different sets of processor and RAM within a LAN, e.g. two laptops at home. (e.g. use RMI, Docker, Kubernetes?)
Level 3: one instance of each of A and B running on different sets of processor and RAM on WAN, e.g. my laptop and a PC at a remote location
Level 4 (yes, may overlap with Level 2 and 3, conceptually): multiple instances of A and B running on different nodes of cloud service such as AWS Cloud and Google Cloud. (e.g. use RMI, Docker, Kubernetes?)
Question 2: if my ultimate goal is as per Level 4, but I'll start development of A and B on my laptop first, similar to level 1, what could be an overall good approach to do this whole development/deployment cycle?
Kubernetes is a good option for it's elasticity: from single host single node to huge deployments.
After your initial stage on minikube you can easily export the yaml of your configurations, in order to run in a bigger local cluster, a hybrid or 100% on the Cloud.
Edit:
Whether Kubernetes platform is suitable or not for running a High Frequency Trading (HFT) like application within it, is yet another topic, and requires opening a seperate thread on SO.
I can only remind here, that Kubernetes design was influenced by Google's Borg system, which in turn was made among to others to handle short-lived latency-sensitive requests (e.g. web search). Check wikipedia to learn more about this topic.
Nowadays applications running on Kubernetes can use natively underlying hardware. For instance, local storage directly attached to workers nodes (like NVMe SSDs drives) or GPU resources, which offers respectively consistent high performance of Disk I/O operations for your applications and can accelerate compute intensive tasks.