A Java gRPC server is running as a container in kubernetes, I am not able to figure out how to check the total number of gRPC requests hits to this server. It isn't equal to the number of successful hits, maybe the server is down and not able to entertain the requests, but hit count would increase anyway.
Any help would be appreciated.
You can build logging into your server using logging via interceptors. This allows collection of data while your server is running, but obviously not when it isn't.
If you implement a TCP liveness probe for your service, Kubernetes can automatically take care of some sources of downtime e.g. your service needing restarted or moved to another node.
This might cut down the cases when you actually need to count failed requests.
You could use a gRPC-aware proxy that sits between your server and the cluster / outside world e.g. nginx.
This would allow you to count all requests regardless of success or failure (assuming the cluster was always running).
You can have kubernetes ingres monitoring and metrics set up using prometheus and Grafana. Your k8s cluster may have already setup this, check with your ops guy
More here
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/monitoring.md
Once your metric is in prometheus, you can set up custom metrics and alerts off of that data using Grafana