Is there Failover Feature In Kubernetes Services?

11/29/2019

I have a kubernetes cluster and 2 replicated databases.I want to be able to connect that databases inside cluster. Databases will be statefulset and have a headless server. Lets say that pod names are D-1 and D-2. D-1 will be master and D-2 will be replica. I want to redirect all my traffic to master (D-1) but if master pod fails it should redirect traffic to replica(D-2). I can not do it on client side so i must find a way in kubernetes. How can i build this structure ? So if i curl to

database-service.default.svc.cluster.local

it must always go to D-1 pod and if D-1 pod is not available it must redirect all traffic to D-2.

-- akuscu
kubernetes
load-balancing

2 Answers

12/3/2019

I thought i can handle with this way(not tested): think about 2 database pods on 2 nodes.If D-2 has a readiness probe depend on D-1 is up(if D1-1 is up the D-2 has readiness probe situation and services mark this pod as unhealty) all the traffic will be redirected to the D-1(master).And if i replicate that databases from hostport i think it could be done.Is it a true approach ?

Also there is cloud native storage solutions.

-- akuscu
Source: StackOverflow

12/3/2019

The answer depends on DB type you are using.

Generally Service is just an abstract way to expose an application running on a set of Pods as a network service.

$  kubectl get svc -o wide
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE    SELECTOR
kubernetes           ClusterIP   10.0.0.1     <none>        443/TCP        6d3h   <none>
db-access            ClusterIP   10.0.5.233   <none>        3306/TCP       3m     app=my_db

Service in that example routes traffic to the Enpoint Slice according to the app=my_db selector.

As you know, K8s evicts failed pods automatically that is why traffic isn't routed to the failed pods. However,

  • K8s loadbalances traffic (through the service) when both DBs are online;
  • there is a delay between the Pod crash and moment it's evicted (so traffic isn't routed to it).

That is why in order to achieve the fail over you have described:

I want to redirect all my traffic to master (D-1) but if master pod fails it should redirect traffic to replica(D-2)

it looks like the "operator" is exactly what is needed here. Operator is not something that is included to K8s, but often provided by the DBVendor.

A good example here is the Oracle MySQL Operator for Kubernetes.

It is a K8s controller that can be installed into any existing K8s cluster. Once installed, it will enable users to create and manage production-ready MySQL clusters using a simple declarative configuration format.

It's feature rich, allows you to manage cluster, perform backups/restores, provides metrics, etc.

Additionally, you can consider using Cloud native distributed databases as @Jonas suggested earlier.

Hope that helps.

-- Nick
Source: StackOverflow