Redirection Stateful pod in Kubernetes

5/8/2019

I am using stateful in kubernetes. I write an application which will have leader and follower (using Go) Leader is for writing and reading. Follower is just for reading.

In the application code, I used "http.Redirect(w, r, url, 307)" function to redirect the writing from follower to leader.

if I use a jump pod to test the application (try to access the app to read and write), my application can work well, the follower can redirect to the leader

kubectl run -i -t --rm jumpod --restart=Never --image=quay.io/mhausenblas/jump:0.2 -- sh

But when I deploy a service (to access from outside).

apiVersion: v1
kind: Service
metadata:
  name: service-name
spec:
  selector:
    app: app-name
  ports:
  - port: 80
    targetPort: 9876

And access to application by this link:

curl -L -XPUT -T /tmp/put-name localhost:8001/api/v1/namespaces/default/services/service-name/proxy/

Because service will randomly select a pod each time we access to the application. When it accesses to leader, it work well (no problem happen), But if it accesses to the follower, the follower will need to redirect to leader, I met this error:

curl: (6) Could not resolve host: pod-name.svc-name.default.svc.cluster.local

What I tested:

  • I can use this link to access when I used jump pod

  • I accessed to per pod, look up the DNS. I can find the DNS name by "nslookup" command

  • I tried to fix the IP of leader in my code. In my code, the follower will redirect to a IP (not a domain like above). But It still met this error:

    curl: (7) Failed to connect to 10.244.1.71 port 9876: No route to host

Anybody know this problem. Thank you!

-- nguyennd
kubernetes

2 Answers

5/8/2019

In order for pod DNS to work you must create headless service:

apiVersion: v1
kind: Service
metadata:
  name: svc-name-headless
spec:
  clusterIP: None
  selector:
    app: app-name
  ports:
  - port: 80
    targetPort: 9876

And then in StatefulSet spec you must refer to this service:

spec:
  serviceName: svc-name-headless

Read more here: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id

Alternatively you may specify what particular pod will be selected by Service like that:

apiVersion: v1
kind: Service
metadata:
  name: svc-name
spec:
  selector:
    statefulset.kubernetes.io/pod-name: pod-name-0
  ports:
  - port: 80
    targetPort: 9876
-- Vasily Angapov
Source: StackOverflow

5/8/2019

When you are accessing your cluster services from outside, a DNS names reserved normally for inner cluster use (e.g. pod-name.svc-name.default.svc.cluster.local) are not recognized by clients (e.g. Web browser).

Context:
You are trying to expose access to PODs, controlled by StatefullSet with service of ClusterIP type

Solution:
Change ClusterIP (assumed by default when not specified) to NodePort or LoadBalancer

-- Nepomucen
Source: StackOverflow