Notifications between Pods in Kubernetes

8/26/2021

The problem:

Websockets and Socket.io allow for rich 2-way asynchronous notifications between client and webserver.

Socket.io between HTML/javascript client and ingress "cookie" routing creates a stateful association between a Pod in a Deployment (let's call these pods of Deployment A) and the HTML/javascript client. Other Pods in other Deployments (let call these Pods of Deployment B and Deployment C) may want to notify the specific Pod in Deployment A of events specific to what Pod A is displaying.

Is there a Kubernetes mechanism to allow this registration and communication between pods?

Generic Setup:

Deployment A, B, C each have multiple replicas.

The pods from A, B, and C can read and update records in a dumb distributed store.

Each pod from A will be responsible for a set of pages (i.e., webserver). The set of pages that a specific pod A is responsible for can change dynamically (i.e., user decides what page to edit). pod A are not stateless since "cookie" control ingress routing and pod A maintains a socket.io to user html/javascript page.

Pods in B or C update components on the pages. If a page component is updated by B/C that is currently being edited by pod in A, the B/C pod must notify the specific pod A of the update.

More than one pod A may be editing the same page.

More detail:

Deployment A is a nodejs express server hosting socket.io from html/javascript clients. Traffic is routed from an ingress using nginx.ingress.kubernetes.io/affinity: "cookie" so the pod hosting a specific client can send unsolicited traffic to the client. i.e., asynchronous two-way traffic.

Deployment B is is a backend for Deployment A. A simple socket.AF_INET is opened from a pod in deployment A to a Service for deployment B. Response from B goes to A and then to the client. So far, so good, everything works, but only tested on 1 node configuration.

Deployment C is a backend to Deployment B. A socket is opened from B to a Service for C. Response from C to B to A to the WebClient works fine (again on a 1 node configuration)

The problem:

Deployment B and C may get requests from other sources to process information that would change the content displayed to the user. I want to update any pod on deployment A that is hosting socket.io to the client displaying this page.

The description/implementation so far does not asynchronously update the pod A unless the user does a full page refresh.

Multiple users may display the same page but be associated via ingress cookies to different deployment A pods.

As it stands now, user 1 only sees updates initiated by user 1, and user 2 only sees updates from user 2. Unless each user does a page refresh.

I want B and C to send updates to all pods in A that are displaying the page being updated.

Solution that does not feel Kubernetes clean:

A pod wanting notifications of changes to components on a page would create a record indicating it's interested in changes to this page containing its pod's IP address, and a good until timestamp.

When a client displays a page, the hosting A pods will update a record in the distributed datastore to indicate it wants updates to components on this page. Every so often, Pod A will update the keep alive time in this records. When the user leaves this page, associated Pod A will remove this record.

Any pod updating records for a page would check this record and open a socket to this other pod to notify it of changes.

An audit would remove any records expired that were not correctly cleanup because of abnormal termination of a pod after it registered its interest in being notified of changes.

Question Restated:

Is this a clean Kubernetes solution or is there something in Kubernetes to make this cleaner?

New to Kubernetes: did I mess up Kubernetes nomenclature anywhere in my question?

-- grabbag
kubernetes
kubernetes-ingress
kubernetes-pod

1 Answer

8/27/2021

There is no native way to reliably communicate between pods of different Deployments. It's kind of possible in case of StatefulSets where all pods have known and stable names.

In your case, assuming I understand your requirements correctly, I would recommend using queue messaging software like RabbitMQ or Kafka

-- p10l
Source: StackOverflow