K
Q

Q: Configuration custom scheduler plugin in Kubernetes

9/20/2019

I'm trying to add my custom scheduler with scheduler plugin in Kubernetes. Due to no detailed example in Kubernetes document (scheduling-framework, 20180409-scheduling-framework)

So I simply made it for testing. But, I got error: unable to recognize "test.yaml": no matches for kind "KubeSchedulerConfiguration" in version "kubescheduler.config.k8s.io/v1alpha1".

How could I configure and use scheduler plugin?

In my case, test configurations are

apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
plugins:
  queueSort:
    enabled:
    - name: test

pluginConfig:
- name: test-

kubectl command

$ kubectl apply -f test.yaml
error: unable to recognize "test.yaml": no matches for kind "KubeSchedulerConfiguration" in version "kubescheduler.config.k8s.io/v1alpha1"

When I created as pod,

$ kubectl create -f test.yaml
$ kubectl logs pod test
E0920 15:55:47.970707       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0920 15:55:47.971808       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:48.962810       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "nodes" in API group "" at the cluster scope
E0920 15:55:48.963476       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:48.964615       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0920 15:55:48.965674       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0920 15:55:48.966800       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0920 15:55:48.967967       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "services" in API group "" at the cluster scope
E0920 15:55:48.969030       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "pods" in API group "" at the cluster scope
E0920 15:55:48.970095       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0920 15:55:48.971150       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0920 15:55:48.972252       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0920 15:55:48.973396       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:49.964805       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "nodes" in API group "" at the cluster scope
E0920 15:55:49.965338       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:49.966313       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0920 15:55:49.967467       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0920 15:55:49.968448       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0920 15:55:49.969534       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "services" in API group "" at the cluster scope
E0920 15:55:49.970775       1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Pod: pods is f^C

Enviroment

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

go

$ go version
go version go1.13 linux/amd64

go module

$ cat go.sum
...

k8s.io/kubernetes v1.16.0 h1:WPaqle2JWogVzLxhN6IK67u62IHKKrtYF7MS4FVR4/E=
k8s.io/kube-scheduler v0.0.0-20190918162820-3b5c1246eb18 h1:WnbesbtFrWlZ8wwgDuP1KN+6E+XLDGzG4ztojMxfHsc=

...
-- yotion
kube-scheduler
kubernetes
plugins

Similar Questions

Is there any REST APIs for native kubectl cmds like apply/delete etc
Is it possible to set the IpcMode for a container with kubernetes?
Is there a sneaky way to run a command before the entrypoint (in a k8s deployment manifest) without having to modify the dockerfile/image?
Pod route in Kubernetes

1 Answer

9/24/2019

The scheduler configuration is not applied to the API server with kubectl apply command, it is passed to the scheduler process with kube-scheduler --config=/path/to/config. Read more about it on Github.

--config is the entry where you should specify which parameters the scheduler will take. From the API’s perspective, that config file is supposed to contain a KubeSchedulerConfiguration object:

 # content of the file passed to "--config"
 apiVersion: kubescheduler.config.k8s.io/v1alpha1
 kind: KubeSchedulerConfiguration
 clientConnection:
   kubeconfig: "/var/run/kubernetes/scheduler.kubeconfig"
 algorithmSource:
   policy:
     file:
       path: "/root/config/scheduler-extender-policy.json"

Refer to this article, which describe how to create custom Kubernetes scheduler and configuration of Scheduler.

-- muscat
Source: StackOverflow