The kubeflow pipelines would make a good fit with your specific use case. The idea is that you containerize all the individual steps that you want to have decoupled, something like: 1/ preprocessing, 2/ training, 3/ serving. Each container is designed so it can take the relevant arguments that you want to modify over time to run different versions of the pipeline.

• For the preprocessing image I would suggest to start from a GPU image with opencv installed that drops the output on Google Cloud Storage.
• For the training you could leverage the google/cloud-sdk:latest image that comes with gcloud command so you just copy over your code and run the ml engine command.
• For serving, you could use ml engine to deploy the model and thus start building the image from again google/cloud-sdk:latest, alternatively you could use the TF serving images that are available off the shelf and you only need to specify the bucket where your saved model is stored and the model name see instructions.

This blog posts describes how to build a similar pipeline.