Spinnaker on Kubernetes, how to configure the redis and cassandra connection?

6/2/2016

I'm currently trying to start spinnaker on our internal kubernetes cluster. The current state is:

$ kubectl get pods
NAME                           READY     STATUS             RESTARTS   AGE
data-cassandra-v000-1azd8      1/1       Running            0          3h
data-redis-master-v000-68jzu   1/1       Running            0          4h
spkr-clouddriver-v000-0xpdl    1/1       Running            0          3h
spkr-deck-v000-xsl4o           1/1       Running            0          3h
spkr-echo-v000-metx4           0/1       CrashLoopBackOff   44         3h
spkr-front50-v000-nqfrx        0/1       Running            29         3h
spkr-gate-v000-ep56r           1/1       Running            0          3h
spkr-igor-v000-svkwb           0/1       CrashLoopBackOff   44         3h
spkr-orca-v000-bdeig           0/1       CrashLoopBackOff   44         3h

I currently have 2.5 issues I'm not sure how to get around:

  • the orca service cannot connect to redis. It gets a connection refused. I assume it tries to connect to localhost (or 0.0.0.0). The logs only show a stackstrace with a java.net.ConnectException. How do I point it to the correct IP? Do I start the redis service first and just figure out where it ends up?
  • similar issue with the echo service connecting to cassandra. I get a: PoolTimeoutException: [host=localhost(127.0.0.1):9160. I assume it wants to try connecting to cassandra as that would be the port. How is this connection supposed to work? Run cassandra and update the ip from what I end up with? Since everything is dynamic I hoped to find some service name relation but I'm not sure what to put in the -local.yml files for the configuration.
  • third issue lies in igor. It complains about an empty githubToken although travis is disabled. I'm going to try just some values once I'm brave enough to restart to whole thing.

Any hints are welcome :)

Thanks a lot!

Update: After @lwander pointed me to check the spinnaker-local.yml file I noticed that file was probably a copy of the one created from default-spinnaker-local.yml. I reverted that and tried again:

$ kubectl get pods
NAME                           READY     STATUS             RESTARTS   AGE
data-cassandra-v000-gxgtv      1/1       Running            0          19m
data-redis-master-v000-y4iug   1/1       Running            0          20m
spkr-clouddriver-v000-xe3pe    1/1       Running            0          13m
spkr-deck-v000-mvs88           1/1       Running            0          13m
spkr-echo-v000-tjnk9           0/1       CrashLoopBackOff   6          13m
spkr-front50-v000-s4r61        0/1       Running            3          13m
spkr-gate-v000-5wz0f           0/1       Running            0          13m
spkr-igor-v000-xqv9g           1/1       Running            0          13m
spkr-orca-v000-ahz47           0/1       CrashLoopBackOff   6          13m

But I still get connection errors, just the hostname is no longer localhost but ending with .spinnaker.svc.cluster.local - not sure if thats also a value I would need to change or one that should work?

$ kubectl logs spkr-echo-v000-tjnk9
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=data-cassandra.spinnaker.svc.cluster.local(data-cassandra.spinnaker.svc.cluster.local):9160, latency=2000(2000), attempts=1]Timed out waiting for connection
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231)
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198)
        at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84)
        at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117)
        at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
        at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:517)
        at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.internalDescribeKeyspace(ThriftKeyspaceImpl.java:268)
        at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.describeKeyspace(ThriftKeyspaceImpl.java:264)
        at com.netflix.scheduledactions.persistence.cassandra.ThriftCassandraDao.createColumnFamily(ThriftCassandraDao.java:81)
        ... 70 more


$ kubectl logs spkr-orca-v000-ahz47
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        ... 21 more
Caused by: java.net.UnknownHostException: data-redis-server.spinnaker.svc.cluster.local
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at redis.clients.jedis.Connection.connect(Connection.java:158)
        ... 44 more


$ kubectl logs spkr-igor-v000-xqv9g
Caused by: java.net.UnknownHostException: spkr-clouddriver.spinnaker.svc.cluster.local: unknown error
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at com.squareup.okhttp.internal.Network$1.resolveInetAddresses(Network.java:29)

Update 2: it seems it not too far away to connect the parts. The fabric8 ui for the kubernetes cluster seems to recognize most of it.

fabric8 overview

-- wemu
kubernetes
spinnaker

0 Answers