I'm using Kubernetes 1.7 and Python Client 2.0. I have a hello world machine learning program MNIST in TensorFlow running under K8 cluster. It has one Worker and one Parameter Server. It is deployed as a kind: Job (in the manifest). The custom scheduler, written in python, watches for pending pods using list_namespaced_pod and schedules them based on the availability of resources. Since it's a stream of events coming in, how can I make sure that all pending pods under one job get scheduled or not? In other words, I don't want to schedule a job partially, either to schedule all pods of the pending jobs or none.

Also, is there a way in Kubernetes to catch/find/watch all events of the same job (i.e. deployed under one manifest file) at a time? I also tried list_namespaced_event but it also reports events one after another. As a result, it is likely to happen that one pod of the job can be scheduled and the latter one can't. A small version of the custom scheduler is available here.

my-mnist.yml file (a smaller version)

---

apiVersion: batch/v1
kind: Job
metadata:
  name: my-ps 
  labels:
    name: my-ps 
    jobName: my-ps-mnist_dist
  namespace: my-namespace
spec:
  template:
    metadata:
      labels:
        name: my-ps 
        jobName: my-ps-mnist_dist
        jobId: 5b2a6cd25b02821468e41571
        manifestFile: my-mnist.yml
        jobTrainingType: distributed
        jobTaskName: "my-ps"
        jobTaskIndex: "0"
        jobWorkerInstances: "1"
      namespace: my-namespace
    spec:
      nodeSelector:
        gpu: "no"
        dlts: "yes"
      containers:
        - name: my-ps
          image: "123.456.789.10:1234/myimg/5b2a6cd25b02821468e41571"
          imagePullPolicy: Always
          tty: true
          stdin: true
          env:
            - name: JOB_TASK_NAME
              value: "ps"
            - name: JOB_ID
              value: "5b2a6cd25b02821468e41571"
            - name: JOB_LD_LIBRARY_PATH
              value: "/usr/local/cuda-9.0/lib64:/usr/lib64/nvidia:/usr/local/cuda-9.0/targets/x86_64-linux/lib"
            - name: JOB_PYTHON_VERSION
              value: "3"


---

apiVersion: batch/v1
kind: Job 
metadata:
  name: my-wkr
  labels:
    name: my-wkr
    jobName: wkr0-mnist_dist
  namespace: my-namespace
spec:
  template:
    metadata:
      labels:
        name: my-wkr
        jobName: wkr0-mnist_dist
        jobId: 5b2a6cd25b02821468e41571
        manifestFile: my-mnist.yml
        jobTrainingType: distributed
        jobTaskName: "worker"
        jobTaskIndex: "0"
        jobWorkerInstances: "1" 
      namespace: my-namespace
    spec:
      nodeSelector:
        gpu: "yes"
        dlts: "yes"
      containers:
        - name: my-wkr
          image: "123.456.789.10:1234/myimg/5b2a6cd25b02821468e41571" 
          imagePullPolicy: Always
          tty: true
          stdin: true
          resources:
            limits:
              alpha.kubernetes.io/nvidia-gpu: 2
          env:
            - name: JOB_TASK_NAME
              value: "worker"
            - name: JOB_TASK_INDEX
              value: "0"
            - name: JOB_ID
              value: "5b2a6cd25b02821468e41571"
            - name: JOB_LD_LIBRARY_PATH
              value: "/usr/local/cuda-9.0/lib64:/usr/lib64/nvidia:/usr/local/cuda-9.0/targets/x86_64-linux/lib"
            - name: JOB_PYTHON_VERSION
              value: "3"