Fail-fast behavior for Eureka client

6/19/2020

It seems that following problem doesn't have common decision and I try to solve it from another side. Microservices infrastructure consists of Spring Boot Microservices with Eureka-Zuul-Config-Admin Servers as service mesh. All Microservices runs inside Docker containers at the Kubernetes platform. Kubernetes monitors application health check (liveness/readyness probes) and redeploy it when health check in down state exceeds liveness probe timeout.

The problem is following - sometimes Microservice doesn't get correct Eureka server address after redeployment. Service discovery registration fails but Microservice continue working with health check 'UP' and dependent Microservices miss it.

Microservices are interdependent and failure of one Microservice causes cascade failure of all dependent Microservices. I don't use Histrix because of some reasons and actually it is not resolve my problem - missed data from failed Microservice just disables entire functionality related to the set of dependent Microservices.

Question: Is it possible to configure something like 'fail-fast' behavior for Eureka client without writing custom HealthIndicator? The actuator health check should be in 'DOWN' state while Eureka client doesn't get 204 successful registration response from Eureka.

Here is an example of how I fix it in code. It has pretty simple behavior - healthcheck goes down 'forever' after exceeding timeout to successful registration in Eureka on start or/and during runtime. The main goal is that the Kubernetes will redeploy Microservice when liveness probe timeout exceeded.

@Component
public class CustomHealthIndicator implements HealthIndicator {

    private static final Logger logger = LoggerFactory.getLogger(CustomHealthIndicator.class);

    @Autowired
    @Qualifier("eurekaClient")
    private EurekaClient eurekaClient;

    private static final int HEALTH_CHECK_DOWN_LIMIT_MIN = 15;

    private LocalDateTime healthCheckDownTimeLimit = getHealthCheckDownLimit();

    @Override
    public Health health() {
        int errCode = registeredInEureka();
        return errCode != 0
                ? Health.down().withDetail("Eureka registration fails", errCode).build()
                : Health.up().build();
    }

    private int registeredInEureka() {
        int status = 0;
        if (isStatusUp()) {
            healthCheckDownTimeLimit = getHealthCheckDownLimit();
        } else if (LocalDateTime.now().isAfter(healthCheckDownTimeLimit)) {
            logger.error("Exceeded {} min. limit for getting 'UP' state in Eureka", HEALTH_CHECK_DOWN_LIMIT_MIN);
            status = HttpStatus.GONE.value();
        }
        return status;
    }

    private boolean isStatusUp() {
        return eurekaClient.getInstanceRemoteStatus().compareTo(InstanceInfo.InstanceStatus.UP) == 0;
    }

    private LocalDateTime getHealthCheckDownLimit() {
        return LocalDateTime.now().plus(HEALTH_CHECK_DOWN_LIMIT_MIN, ChronoUnit.MINUTES);
    }
}

Is it possible to do the same by just configuring Spring components?

-- Kirill
kubernetes
microservices
netflix-eureka
spring-boot

0 Answers