Hystrix-dashboard not working with Kubeflix on Kubernetes

4/3/2017

I'm deploying microservices to Kubernetes with Spring Boot, Fabric8's Spring-Cloug-Kubernetes and I now want to have a Hystrix Dashboard, provided by Fabric8 Kubeflix.

I have set labels on my deployments :

metadata : 
    labels:
        hystrix.cluster: default
        hystrix.enabled: true
spec:
    template:
        metadata : 
            labels:
                hystrix.cluster: default
                hystrix.enabled: true

And it is on my pods too :

metadata : 
    labels:
        hystrix.cluster: default
        hystrix.enabled: true

In the turbine-server pod logs I have :

2017-03-31T15:23:10.514696068Z 2017-03-31 15:23:10.514  INFO [turbine-server,,,] 1 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Found hosts that have been previously terminated: 0
2017-03-31T15:23:10.514700568Z 2017-03-31 15:23:10.514  INFO [turbine-server,,,] 1 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Hosts up:0, hosts down: 0

And its /discovery endpoint displays :

Hystrix Endpoints:

turbine-server application.yml :

spring:
  application:
    name: turbine-server
turbine:
  instanceUrlSuffix: :80/hystrix.stream
  aggregator:
    clusterConfig: default
InstanceDiscovery:
  impl: io.fabric8.kubeflix.turbine.TurbineDiscovery

On my microservices I just have a

@EnableCircuitBreaker
@EnableHystrix

in their main Application classes.

I set port 80 as turbine suffix because I have Kubernetes services exposing port 80 of pods :

spec: 
    ports: [
        name: default   
        protocol:   TCP,
        port:   80,
        targetPort: 8080,
        nodePort:   32193
    ],
    clusterIP:  10.0.72.62,
    type:   NodePort

When I do a /health on my service :

"status": "UP",
"hystrix": {
    "status": "UP"
},

And the /hystrix.stream :

data: {"type":"HystrixCommand","name":"getLabel","group":"LabelController","currentTime":1491222462325,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackMissing":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":1000,"propertyValue_executionTimeoutInMilliseconds":1000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"LabelController"}
data: {"type":"HystrixThreadPool","name":"LabelController","currentTime":1491222462325,"currentActiveCount":0,"currentCompletedTaskCount":2,"currentCorePoolSize":10,"currentLargestPoolSize":2,"currentMaximumPoolSize":10,"currentPoolSize":2,"currentQueueSize":0,"currentTaskCount":2,"rollingCountThreadsExecuted":0,"rollingMaxActiveThreads":0,"rollingCountCommandRejections":0,"propertyValue_queueSizeRejectionThreshold":5,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"reportingHosts":1}

The hystrix-dashboard receive the following turbine.stream :

{"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1491222578286}

Versions :

Spring Boot : 1.4.1.RELEASE
io.fabric8.kubernetes-client : 1.4.14
io.fabric8.kuberflix.turbine-discovery : 1.0.28
Spring cloud Neflix : 1.2.4.RELEASE
Netflix hystrix : 1.5.6

What is wrong with my turbine-server ? Why is it not detecting hystrix.stream from my microservices ?

-- Michaël L
fabric8
hystrix
kubernetes
spring-cloud-netflix
turbine

1 Answer

4/3/2017

Ok it works, I had to disable the default Eureka discovery :

ribbon.eureka.enabled: false
eureka.client.enabled: false

so that the Kubernetes one is used.

And the instanceUrlSuffix still need to be :8080/hystrix.stream as it takes the internal k8s IP of the pods.

-- Michaël L
Source: StackOverflow