I have a bunch of microservices running in a kubernetes cluster where each microservice implements a basic health check over HTTP.
e.g for the endpoint /health each service will return a HTTP response 200 if that particular service is currently healthy or some other HTPP 4xx / 5xx code (and possible additional info) if not healthy.
I see Kubernetes has its own built in concepth of a HTTP health check https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-http-request
Unfortunatley its not quite what I want. I like to be able to trigger an alert (and record the state of the health check request) in some database so I can quickly check what state all my services are in as well as alerting on any services in an unhealthy state.
I'm wondering are there existing tools or approaches in Kubernetes I should use for this sort of thing? Or will need to build some custom solution for this.
Was considering having a general "HealthCheck" service which each microservice would register with when started. That way the "HealthCheck" service would monitor the health of each service as well as trigerring alerts for any issues it finds.
I would caution against trying to build your own in-house monitoring solution. There are considerable drawbacks to that approach.
If all you need is external service HTTP health checks, then many existing monitoring solutions will do fine. You can either install a traditional IT solution like Zabbix or Nagios. Or use a SAS like Datadog and others.
There are also blackbox extensions for Prometheus, which is very popular among K8s users.
Many of these options have a learning curve of some steepness.