Exposing cassandra cluster on minikube to access externally

6/18/2019

I'm trying to deploy a cassandra multinode cluster in minikube, I have followed this tutorial Example: Deploying Cassandra with Stateful Sets and made some modifications, the cluster is up and running and with kubectl I can connect via cqlsh, but I want to connect externally, I tried to expose the service via NodePort and test the connection with datastax studio (192.168.99.100:32554) but no success, also later I want to connect in spring boot, I supose that I have to use the svc name or the node ip.

All host(s) tried for query failed (tried: /192.168.99.100:32554 (com.datastax.driver.core.exceptions.TransportException: [/192.168.99.100:32554] Cannot connect))

[cassandra-0] /etc/cassandra/cassandra.yaml

rpc_port: 9160
broadcast_rpc_address: 172.17.0.5
listen_address: 172.17.0.5
# listen_interface: eth0
start_rpc: true
rpc_address: 0.0.0.0
# rpc_interface: eth1
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "cassandra-0.cassandra.default.svc.cluster.local"

Here is minikube output for the svc and pods

$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
cassandra    NodePort    10.102.236.158   <none>        9042:32554/TCP   20m
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP          22h

$ kubectl get pods -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
cassandra-0   1/1     Running   0          20m   172.17.0.4   minikube   <none>           <none>
cassandra-1   1/1     Running   0          19m   172.17.0.5   minikube   <none>           <none>
cassandra-2   1/1     Running   1          19m   172.17.0.6   minikube   <none>           <none>

$ kubectl describe service cassandra
Name:                     cassandra
Namespace:                default
Labels:                   app=cassandra
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"cassandra"},"name":"cassandra","namespace":"default"},"s...
Selector:                 app=cassandra
Type:                     NodePort
IP:                       10.102.236.158
Port:                     <unset>  9042/TCP
TargetPort:               9042/TCP
NodePort:                 <unset>  32554/TCP
Endpoints:                172.17.0.4:9042,172.17.0.5:9042,172.17.0.6:9042
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

$ kubectl exec -it cassandra-0 -- nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.5  104.72 KiB  256          68.1%             680bfcb9-b374-40a6-ba1d-4bf7ee80a57b  rack1
UN  172.17.0.4  69.9 KiB   256          66.5%             022009f8-112c-46c9-844b-ef062bac35aa  rack1
UN  172.17.0.6  125.31 KiB  256          65.4%             48ae76fe-b37c-45c7-84f9-3e6207da4818  rack1

$ kubectl exec -it cassandra-0 -- cqlsh 
Connected to K8Demo at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> 

cassandra-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  type: NodePort
  ports:
  - port: 9042
  selector:
    app: cassandra

cassandra-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: cassandra:3.11
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: CASSANDRA_START_RPC
            value: "true"
          - name: CASSANDRA_RPC_ADDRESS
            value: "0.0.0.0"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /var/lib/cassandra
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
  type: pd-standard
-- jhuamanchumo
cassandra
kubernetes
minikube

2 Answers

11/24/2019

If we just want to connect cqlsh, what you neeed is following command

kubectl exec -it cassandra-0 -- cqlsh

On the other hand, if we want to connect from external point, command can be used to get cassandra url (I use DBever to connect cassandra cluster)

minikube service cassandra --url
-- Olcay Tarazan
Source: StackOverflow

6/21/2019

Just for anyone with this problem: After reading docs on datastax I realized that DataStax Studio is meant for use with DataStax Enterprise, for local development and the community edition of cassanda I'm using DataStax DevCenter and it works.

For spring boot (Cassandra cluster running on minikube):

spring.data.cassandra.keyspacename=mykeyspacename
spring.data.cassandra.contactpoints=cassandra-0.cassandra.default.svc.cluster.local
spring.data.cassandra.port=9042
spring.data.cassandra.schemaaction=create_if_not_exists

For DataStax DevCenter(Cassandra cluster running on minikube):

ContactHost = 192.168.99.100
NativeProtocolPort: 300042

Updated cassandra-service

# ------------------- Cassandra Service ------------------- #
apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  type: NodePort
  ports:
  - port: 9042
    nodePort: 30042
  selector:
    app: cassandra
-- jhuamanchumo
Source: StackOverflow