My team is looking for a way to run Spark jobs that use the Tensorflow library on Kubernetes. We use the Spark Operator to run Spark on Kubernetes idiomatically.

How should I go about creating a pod with the Spark job (PySpark + TF) and have it work with the Spark Operator k8s resources?

I've explored Horovod, an open source Deep Learning Framework from Uber. We don't use GPUs for training and Horovod seems to be more suited for those operations than what we're aiming for.