K
Q

Kubernetes Python Client error create_namespaced_binding: (409) Reason: Conflict

6/19/2018

I'm using k8 v1.7 and Python Client v2.0. My custom scheduler detects a pending pod and schedules it successfully. However, after assigning a pod to a node, it complains that the pod is already assigned to a node though it is just assigned by the scheduler itself. Is this something to be concerned? Or how can I fix this?

Error Message

create_namespaced_binding: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 19 Jun 2018 16:14:57 GMT', 'Content-Length': '289', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on pods/binding \"ps0-16-r935x\": pod ps0-16-r935x is already assigned to node \"blipp65\"","reason":"Conflict","details":{"name":"ps0-16-r935x","kind":"pods/binding"},"code":409}

scheduler.py

from kubernetes import client, config, watch
from kubernetes.client.rest import ApiException
config.load_kube_config()
v1 = client.CoreV1Api()

scheduler_name = 'my-custom-scheduler-v1'

def nodes_available():
    ready_nodes = []
    for n in v1.list_node().items:
        for status in n.status.conditions:
            if status.status == 'True' and status.type == 'Ready':
                ready_nodes.append(n.metadata.name)
    return ready_nodes


def scheduler(name, node, namespace='default'):
    body = client.V1Binding()

    target = client.V1ObjectReference()
    target.kind = 'Node'
    target.apiVersion = 'v1'
    target.name = node

    meta = client.V1ObjectMeta()
    meta.name = name

    body.target = target
    body.metadata = meta

    return v1.create_namespaced_binding_binding(name, namespace, body)


def main():
    w = watch.Watch()
    for event in w.stream(v1.list_namespaced_pod, 'default'):
        if event['object'].status.phase == 'Pending' and event['object'].spec.scheduler_name == scheduler_name:
            print "Pending Found"
            try:
                res = scheduler(event['object'].metadata.name,random.choice(nodes_available()))
                print "success"
            except Exception as a:
                print ("Exception when calling CoreV1Api->create_namespaced_binding: %s\n" % a)

POD YML file

apiVersion: v1
kind: Pod
metadata:
  name: shoeb-pod
spec:
  schedulerName: my-custom-scheduler-v1
  containers:
  - name: redis
    image: redis

Updated on 2019-06-03

I just added the updated main method (according to @VAS's answer, thanks) to find the right PENDING pod that has not been scheduled yet. Please see my answer.

-- Abu Shoeb
api
http-headers
kubernetes
python
scheduling

Similar Questions

kube-apiserver high RAM usage
See ALS persistent volume data in Storage explorer
Do initializers (initializerConfiguration) work on k8s 1.10?
Service account role bindings not working for API access

2 Answers

6/4/2019

Here is the updated main method (according to @VAS's answer, thanks) to find the right PENDING pod that has not been scheduled yet.

def main():
    w = watch.Watch()
    for event in w.stream(v1.list_namespaced_pod, 'default'): # default == namespace name
        # All pending pods have 3 states (not scheduled, scheduled, initialized but not ready yet)
        # We look for NOT SCHEDULED pod and conditions==None
        if event['object'].status.phase == 'Pending' and event['object'].status.conditions == None and event['object'].spec.scheduler_name == CUSTOM_SCHEDULER_NAME:
            print "Pending and Not Scheduled POD Found "+event['object'].metadata.name
            try:
                res = scheduler(event['object'].metadata.name,random.choice(nodes_available())) # nodes_available() returns all available nodes
                print "success"
            except Exception as a:
                print ("Exception when calling CoreV1Api->create_namespaced_binding: %s\n" % a)
-- Abu Shoeb
Source: StackOverflow
6/21/2018

When a pod is created, the scheduler gets three "Pending" events:

  1. Pod is not scheduled yet ('node_name': None, 'status': {'conditions': None,...})
  2. Pod is scheduled ('node_name': 'some_node_name','status': {'conditions': [...,'status': True, 'type':'PodScheduled'],...} )
  3. Pod is initialized but not ready yet ('node_name': 'minikube','status': {'conditions': [...,'status': True, 'type':'Initialized'], ... ,'status': False, 'type':'Ready']} )

Therefore, your custom scheduler should bind pod to the node on the first event, check the status of the pod and make sure it is scheduled when the second event appears, and then check if the pod is initialized when the third event appears.

If something goes wrong, the scheduler may need to take into account the previous errors and probably try to schedule the pod to different nodes.

In your case, your scheduler threats all three events like the first one and tries to schedule the pod again and again. That's why you see that "pod xxx is already assigned to node yyy" error.

-- VAS
Source: StackOverflow