I am using Kubeflow fairing to train a TensorFlow model on Kubernetes. The training succeeds but now I want to serve a prediction endpoint.
How can I retrieve the saved TensorFlow session (weights, biases etc.) from the training step so that I can do this? At the moment the result of the training step is saved inside the Docker container running on the Kubernetes cluster.
I had misunderstood the scope of Kubeflow fairing - at the time of writing it doesn't support copying the trained model from the fairing job to where the code was run from, nor is this necessarily desirable.
I instead used the Minio instance provisioned by Kubeflow to store and retrieve tarballs of trained models.