So, i am trying to connect to my scylla instance in GKE with cqlsh by opening a connection with Kubectl. I am however stumbling into some wierd issues i cannot get my head around.
I am running scylla on GKE, it is basicly a cassandra knockoff that is supossed to run way faster than cassandra itself. To access scylla, i want to be able to connect to the db with the kubectl port-forward command so i can hook up external tools such as table plus. When i run kubectl port-forward pod/scylla-0 -n scylla 9042 i expect the port to be accessible from my local machine, however, when i try to connect with cqlsh localy i get these error messages:
from clqsh: Connection error: ('Unable to connect to any servers', {'127.0.0.1': ConnectionShutdown('Connection to 127.0.0.1 was closed',)})
from kubectl: E0520 17:12:12.522329 51 portforward.go:400] an error occurred forwarding 9042 -> 9042: error forwarding port 9042 to pod <some id>, uid : exit status 1: 2019/05/20 15:12:12 socat[998972] E connect(5, AF=2 127.0.0.1:9042, 16): Connection refusedi've also tried to forward the service directly to similar results
What i personaly believe is the wierd part of this is that when i expose scylla with a loadbalancer, i can connect to it perfectly fine, i can also use JConsole when i forward the JMX port for scylla, which is why i am having so many headaches over this.
The reason it is failing is because port-forward only binds to localhost (127.0.0.1). If you go inside the pod with
kubectl exec -ti pod/scylla-0 -n scylla -- /bin/bash
yum install net-tools -y
netstat -nlup | grep scylla You will notice that scylla is binding to the container pod IP and not 127.0.0.1 so you get a connection refused.
Try: kubectl port-forward --address your_pod_ip pod/mypod 9042:9042
Alternatively: kubectl port-forward --address your_service_ip service/myservice 9042:9042
I didn't try it myself but I think it can work. Please let me know.