Unable to run Cassandra on Kubernetes

5/5/2016

I'm trying to run Cassandra on kubernetes.

One thing I understood is Cassandra is trying to access kubernetes api server on port 443 (Secure Connection) but I'm running api server on non secure connection port 8080.

Also there is Has no permission to create /cassandra_data/data directory error

Error (Cassandra pod log):

INFO  13:04:02 Getting endpoints from https://10.100.0.1:443/api/v1/namespaces/default/endpoints/cassandra
WARN  13:04:11 Request to kubernetes apiserver failed
java.io.IOException: Server returned HTTP response code: 401 for URL: https://10.100.0.1:443/api/v1/namespaces/default/endpoints/cassandra
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1628) ~[na:1.7.0_95]
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) ~[na:1.7.0_95]
    at io.k8s.cassandra.KubernetesSeedProvider.getSeeds(KubernetesSeedProvider.java:143) ~[kubernetes-cassandra.jar:na]
    at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:659) [apache-cassandra-2.1.13.jar:2.1.13]
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:136) [apache-cassandra-2.1.13.jar:2.1.13]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:168) [apache-cassandra-2.1.13.jar:2.1.13]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.13.jar:2.1.13]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.13.jar:2.1.13]
INFO  13:04:11 JVM vendor/version: OpenJDK 64-Bit Server VM/1.7.0_95
WARN  13:04:11 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  13:04:11 Heap size: 526385152/526385152
INFO  13:04:11 Code Cache Non-heap memory: init = 2555904(2496K) used = 795136(776K) committed = 2555904(2496K) max = 50331648(49152K)
INFO  13:04:11 Eden Space Heap memory: init = 83886080(81920K) used = 79390336(77529K) committed = 83886080(81920K) max = 83886080(81920K)
INFO  13:04:11 Survivor Space Heap memory: init = 10485760(10240K) used = 0(0K) committed = 10485760(10240K) max = 10485760(10240K)
INFO  13:04:11 CMS Old Gen Heap memory: init = 432013312(421888K) used = 0(0K) committed = 432013312(421888K) max = 432013312(421888K)
INFO  13:04:11 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 19488984(19032K) committed = 21757952(21248K) max = 174063616(169984K)
INFO  13:04:11 Classpath: /kubernetes-cassandra.jar:/etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.13.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.13.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.9.2.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar
INFO  13:04:11 JVM Arguments: [-ea, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms512M, -Xmx512M, -Xmn100M, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -XX:+UseCondCardMark, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=, -Dcassandra-foreground=yes]
WARN  13:04:13 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  13:04:13 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
ERROR 13:04:20 Directory /cassandra_data/data doesn't exist
ERROR 13:04:20 Has no permission to create /cassandra_data/data directory

ReplicationController:

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  replicas: 2
  selector:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      containers:
        - command:
            - /run.sh
          resources:
            limits:
              cpu: 0.1
          env:
            - name: MAX_HEAP_SIZE
              value: 512M
            - name: HEAP_NEWSIZE
              value: 100M
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          image: gcr.io/google-samples/cassandra:v8
          name: cassandra
          ports:
            - containerPort: 9042
              name: cql
            - containerPort: 9160
              name: thrift
          volumeMounts:
            - mountPath: /cassandra_data
              name: data
      volumes:
        - name: data
          emptyDir: {}
-- sravis
cassandra
cassandra-2.0
cassandra-2.1
kubernetes

1 Answer

5/6/2016

You changed cassandra default data directory so first create data directory . Give permission to write and read to created new data directory or run cassandra as root . if you set credential for cassandra then provide user name and password also .

-- mahendra singh
Source: StackOverflow