I am trying to deploy an application with load-balancing within kubernetes
below are my intended deployment diagram
ideally, the application is deployed by a set of pods using k8s deployment with type of "backend"
normally, user instances are stored in the archive. and are restored into one of the pods dynamically upon request, stay there for a TTL time (say 30 minutes), and deleted and backuped into the archive.
ideally, the load balance is deployed by a set of pods using k8s deployment with type of "frontend".
ideally, the frontend is configured as layer7 session sticky with "sticky = host". the host equals the UID of a backend pod
an user requests the service by a SOAP message, which contains parameters "host" and "user" in its body.
when a SOAP message reaches the frontend, the "host" value is extracted from the message body.
if the "host" value is valid, the SOAP message is forwarded to the corresponding backend pod (whose UID equals the host value). otherwise, a random backend pod is assigned.
(processing here upon is application specific) In a backend pod, the application checks the availability of the user instance by the value of "user".
if already existed, just use it; otherwise, try to restore from the archive; if restoring failed(new user), create a new user instance.
I searched around, and did not find any similar examples. especially layer7 session sticky configuration, and the implementation of custom acquiring of sticky value from the incoming message body.
This sounds like a use-case where you are doing authentication through the front-end loadbalancer. Have you looked at Istio and Ambassador. Seems like Istio and Envoy could provide the service mesh to route the requests to the pods. Then you would have to write a custom plugin module into Ambassador to create this specific routing and authentication mechanism that you are seeking.
Example of Ambassador custom authentication service: https://www.getambassador.io/user-guide/auth-tutorial
https://www.getambassador.io/user-guide/with-istio
This custom sticky session routing can also be done using other API gateways but still using Istio for routing to the different pods. However it would be best if the pods are defined as separate services in order to have easier segmentation by the API gateway (Ambassador, Kong, Nginx) based on the parameters of the message body.