My organization operates a fair number of Spring-boot microservices which commonly use two level of spring-boot-actuator style health-checks: one to inform the load balancers that the service is or isn't available (directly analogous to a liveness probe in Kubernetes), and a separate detailed health-check available over HTTP to allow more fine-grained alerting on the application being in some kind of suboptimal state. Of course, we also record application metrics as a time series and monitor and alert on that, but this is a way to define per application what a basic assessment of "health" is and signal that investigation needs to take place.

For example, perhaps a data set the application relies on is stale, or a dependency is responding slowly. These are conditions that need not take the instance of the service out of load balancing, but should be investigated and resolved.

As we're preparing to migrate many of these services into Kubernetes, we're not seeing a directly analogous convention for that "detailed health-check" pattern that doesn't affect the liveness probe or readiness probe. Has anyone found a good pattern for handling this? We're more than willing to work around adapting the interface to the clustering system, but would like to avoid needing to rework the entire logical approach of our monitoring.