Here is the scenario:
Running a 10-node Spark cluster in a K8s environment (eks). I want customer A uses the first 5 nodes (node1,2,3,4,5) and customer B uses the next 5 nodes all the time.
I don't think K8s Affinity can help me here because Spark Scheduler has mind of it's own.
Spark node != kubernetes node so the (anti)affinity that gives hints to k8s scheduler how to schedule particular pods is out of the question.
Can't you just deploy two standalone spark clusters and give customer A access to the first cluster (say, 1 master and 5 workers) and similarly for the customer B?
If this scenario is something you would like to try, check also my https://github.com/radanalyticsio/spark-operator