I want to create a cloud service, where clients create services dynamically.. A service is basically one pod running an API container, let's say a blog and is accessed through ingress-controller specific url path. Pods live as long as the client is paying of course they are not temporary jobs, they have access to DB and a storage volume.
How do I create and maintain these services dynamically, which means be able to spin them up again in case of node failure.
Thanks
Creating and using single pod replica is not good choice and it will not allow you to take full advantage of k8 features(Zero Down time, High Availability, Scalebility etc)
Consider below points while designing your system
If you consider above points, k8 will take care of availability of your service in case of node or pod failure.
Seeing your requirements, the first thing that comes to my mind, is to use Helm as your middle-ware:
Tool that streamlines installing and managing Kubernetes applications.
You treat each Helm release (instance of Kubernetes application installation) as per customer service, ideally in separate namespace to isolate customer environments. You could use also Helm in client mode only, to generate necessary manifest files, which later on you apply with a simple command (kubectl apply -f $HOME/cust001_svc1.yaml). Information like number of replicas, whether to expose your service via Ingress resource, or whether to use StatefulSet, are all sewed in helm charts.
At least as a proof concept should do in my opinion.
Please check also the concept of Operator in Kubernetes ecosystem.