Consul has a TTL health check which status should be updated periodically over the HTTP interface. From akka.net microservices we were performing the GET request to the registered Consul Service endpoints to reset the TTL timer and staying life at Consul Service dashboard.
Does Kubernetes have something similar to it? Not the liveness/readiness probe that performing requests to pod_ip:port, but waiting for the request from the running app. For example, we want to monitor not just that AKKA app running on some port, but to make sure that each actor in the actor system is healthy.
Thanks!
Kubernetes wants to probe the application (with liveness and readiness probes), while the application wants to send its TTL heartbeats to signal liveness to, say, a Consul agent.
One way to reconcile both health check strategies could be a special sidecar health check server running inside the app's pod. Such a sidecar server would sit between the app and the kubelet, and would handle the TTL heartbeats of the app to update its internal state, noting if the app is still alive. As long as that is the case, it would reply with a 200 OK to the HTTP probes of Kubernetes. Else it would reply with a code outside the 200-300 range to Kubernetes to signal that the app is unhealthy.
The Consul agent itself could serve as part of such a sidecar health-check server. Its HTTP health check API returns the status of the app's TTL-liveness as a JSON object. All that needs to be done is to translate the status into an appropriate HTTP return code. But using Consul agent is totally optional: a sidecar could of course handle the TTL heartbeats itself.