I have a Cassandra cluster running in Kubernetes on AWS (using the incubator Helm chart). When I expose the nodes with a load balancer, Datastax is unable to directly connect to the nodes in the way it wants because it tries to use the IPs internal to the Kubernetes network. But it can still successfully read and write through the loadbalancer. This is not recommended however so I am looking for a proper solution.
Is there a way to have datastax connect properly with this setup? Would it require changing the cassandra.yaml inside the container (yuck)?
You can create a Kubernetes service to expose the pods in your Deployment or StatefulSet. The service could be either a NodePort or LoadBalancer port so that you have external access.
You will also have to add a Selector on both your service and your Deployment/StatefulSet so that your service can attach the right Endpoints
First of all. You can't use LoadBalancer for this purpose.
Set up the service like so:
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
type: NodePort
ports:
- port: 9042
nodePort: 30042
selector:
app: cassandraThe nodePort will be available outside the Kubernetes cluster. The nodePort has to be between 30000-32767.
From datastax driver, connect to K8S_NODE_IP:NODEPORT (not the pod ip).
One option is to use an address translator at the driver:
https://docs.datastax.com/en/developer/java-driver/3.5/manual/address_resolution/
However note that you will loose driver side features like token aware load balancing etc.
The best approach is to provide direct connectivity from your app to all the C* nodes so it can establish connections to each node.