Kubernetes cannot access Cassandra within same namespace

5/2/2019

I get

All host(s) tried for query failed (tried: 10.244.0.72/10.244.0.72:9042 (com.datastax.driver.core.exceptions.TransportException: [10.244.0.72/10.244.0.72:9042] Channel has been closed))

when trying to access Cassandra within the same namespace. Although when I forward ports it works ok from localhost. keyspace is created successfully.

kubectl port-forward cassandra1-0 9042:9042

My yaml

apiVersion: v1
kind: Service
metadata:
  name: cassandra1
  labels:
    app: cassandra1
spec:
  ports:
  - name: "cql"
    protocol: "TCP"
    port: 9042
    targetPort: 9042
  - name: "thrift"
    protocol: "TCP"
    port: 9160
    targetPort: 9160
  selector:
    app: cassandra1
  type: NodePort
---  
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra1
  labels:
    app: cassandra1
spec:
  serviceName: cassandra1
  replicas: 1
  selector:
    matchLabels:
      app: cassandra1
  template:
    metadata:
      labels:
        app: cassandra1
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra1
        image: gcr.io/google-samples/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        - containerPort: 9160
          name: thrift  
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra1-0.cassandra1.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "cassandra1"
          - name: CASSANDRA_DC
            value: "DC1-cassandra1"
          - name: CASSANDRA_RACK
            value: "Rack1-cassandra1"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra1-data
          mountPath: /cassandra1_data
  volumeClaimTemplates:
  - metadata:
      name: cassandra1-data
      namespace: default
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

Cassandra starts with following properties:

Starting Cassandra on 10.244.0.72
CASSANDRA_CONF_DIR /etc/cassandra
CASSANDRA_CFG /etc/cassandra/cassandra.yaml
CASSANDRA_AUTO_BOOTSTRAP true
CASSANDRA_BROADCAST_ADDRESS 10.244.0.72
CASSANDRA_BROADCAST_RPC_ADDRESS 10.244.0.72
CASSANDRA_CLUSTER_NAME cassandra1
CASSANDRA_COMPACTION_THROUGHPUT_MB_PER_SEC
CASSANDRA_CONCURRENT_COMPACTORS
CASSANDRA_CONCURRENT_READS
CASSANDRA_CONCURRENT_WRITES
CASSANDRA_COUNTER_CACHE_SIZE_IN_MB
CASSANDRA_DC DC1-cassandra1
CASSANDRA_DISK_OPTIMIZATION_STRATEGY ssd
CASSANDRA_ENDPOINT_SNITCH SimpleSnitch
CASSANDRA_GC_WARN_THRESHOLD_IN_MS
CASSANDRA_INTERNODE_COMPRESSION
CASSANDRA_KEY_CACHE_SIZE_IN_MB
CASSANDRA_LISTEN_ADDRESS 10.244.0.72
CASSANDRA_LISTEN_INTERFACE
CASSANDRA_MEMTABLE_ALLOCATION_TYPE
CASSANDRA_MEMTABLE_CLEANUP_THRESHOLD
CASSANDRA_MEMTABLE_FLUSH_WRITERS
CASSANDRA_MIGRATION_WAIT 1
CASSANDRA_NUM_TOKENS 32
CASSANDRA_RACK Rack1-cassandra1
CASSANDRA_RING_DELAY 30000
CASSANDRA_RPC_ADDRESS 0.0.0.0
CASSANDRA_RPC_INTERFACE
CASSANDRA_SEEDS cassandra1-0.cassandra1.default.svc.cluster.local
CASSANDRA_SEED_PROVIDER org.apache.cassandra.locator.SimpleSeedProvider
changed ownership of '/cassandra_data/data' from root to cassandra
changed ownership of '/cassandra_data' from root to cassandra

In my application that runs in the same namespace i tried setting cassandraport to 9042 and host to:

10.240.0.4 (hostIP)
10.244.0.72 (podIP)
cassandra1   (name of the service)
cassandra1.default
cassandra1.default.svc.cluster.local
cassandra1-0.cassandra1.default.svc.cluster.local
_cql._tcp.cassandra1.default.svc.cluster.local

I also tried different types of a service: headless, ClusterIP, NodePort

Does anybody has ANY ideas what is wrong or what else can i try to get this to work?

-- rigby
cassandra
kubernetes
kubernetes-service

0 Answers