I have a setup of 3 Rest-Proxy(Kafka-rest) instances in Kubernetes.

https://docs.confluent.io/current/kafka-rest/docs/index.html

https://github.com/confluentinc/kafka-rest

I have added a NetScaler Ingress on top of the deployment.

I see in Citrix NetScaler docs that the NetScaler provides persistence configurations for session stickiness.

Citrix Docs for NetScaler Persistence

In the Kafka-Rest docs it's mentioned:

REST Proxy Clusters and Load Balancing - The REST Proxy is designed to support multiple instances running together to spread load and can safely be run behind various load balancing mechanisms (e.g. round robin DNS, discovery services, load balancers) as long as instances are configured correctly.

And

Although the proxy does not have any persistent state, it is stateful because consumer instances are associated with specific proxy instances.

However, I have tried all possible combinations from the configurations. But still some of my data consume requests in Rest-Proxy are failing.

I get a HTTP 404 Error.

Can somebody guide me, on what kind of session stickiness is needed by Rest-Proxy and how can I implement that in NetScaler in kubernetes.

UPDATE:

I also found this text, which states that we should use the "base_uri" returned by the create consumer rest call. Now my question is, should we use it explicitly in the remaining calls for consuming data instead of the Load Balancer URL, or should the stickiness be configured in the load balancer to take into account the returned URL and send the following requests accordingly (if that's even possible)

If you run more than one instance of the proxy you should provide some load balancing mechanism. The simplest approaches use round-robin DNS or a discovery service to select one instance per application process at startup, sending all traffic to that instance. You can also use an HTTP load balancer, but individual instances must still be addressable to support the absolute URLs returned for use in consumer read and offset commit operations.