I have a java multi threaded main program application that dequeues messages off an event hub (Azure EH is a partitioned messaging system along the lines of Kafka / Amazon Kinesis ) and processes them.
This is deployed on AKS / Kubernetes, for HA I have a few replicas set up. To make it complete, I would like to have a health check probe added so that we could self heal the listener.
Few options that I have been using are
a) JMX based stats and see if it can be of use and monitor general JVM stats
b) Create a small socket application to use a TCP probe, need to check how reliable this is and what security needs to be in place (perhaps just within the cluster should be okay)
c) Embed a small HTTP server and expose an HTTP endpoint (least preferred as it adds overheads to an application that I want to keep small otherwise)
Questions :
a) Is there a suggestion or best practice that is recommended for deploying listener applications like these. Is it okay to use ReplicaSet for HA for this or is there a different "kind" in k8s suited for this
b) Are there any drop in health checks (that I can extend for probes for java main applications). How are these sorts of application health stats monitored in general?
Regards
a) Is there a suggestion or best practice that is recommended for deploying listener applications like these. Is it okay to use ReplicaSet for HA for this or is there a different "kind" in k8s suited for this
A ReplicaSet ensures that a specified number of pod replicas are running at any given time. However, a Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features. Therefore, I recommend using Deployments instead of directly using ReplicaSets, unless you require custom update orchestration or don’t require updates at all. This actually means that you may never need to manipulate ReplicaSet objects: use a Deployment instead, and define your application in the spec section.
b) Are there any drop in health checks (that i can extend for probes for java main applications). How are these sort of application health stats monitored in general ?
Kubernetes provides necessary features for health checks via HTTP, Command, TCP types of Probes. This health checks will provide self healing for your pods.
TCP probes come in handy if you have a scenario where HTTP probes or command probe don’t work well. For example, a gRPC or FTP service is a prime candidate for this type of probe
You need to choose which one best for your use-case. I will also suggest having initial probing delay because liveness probe failure causes the pod to restart. You need to make sure the probe doesn’t start until the app is ready. Otherwise, the app will constantly restart and never be ready.I recommend using the p99 startup time as the initialDelaySeconds, or just take the average startup time and add a buffer. As your app's startup time gets faster or slower, make sure you update this number
Also you should use ingress for north south traffic and you can configure load balancing and health-check in ingress. When this health check fails ingress will stop sending traffic to your backend pod.