We have a Flask server, which makes gRPC calls to a Tensorflow Serving POD(which uses Official tf-serving docker image) that serves around 20 tensorflow models. Both Flask microservice and Tf-Serving microservice are deployed on Kubernetes. For 0.5-1% of the requests made from Flask to Tf-Serving, Flask server is throwing an RpcError - StatusCode.DEADLINE_EXCEEDED

There is a similar issue on github: https://github.com/tensorflow/serving/issues/1265 .Please help me to resolve this issue.