K
Q

Custom Scheduler leaves pod in pending Kubernetes cluster

4/26/2019

I deploy a custom scheduler after following instructions step by step like mentioned in Kubernetes Documentation

Here's [a link] (https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/)

Pods I specify should be scheduled using the scheduler that I deployed "my-scheduler" leaves in Pending.

Kubectl version : -Client: v1.14.1
                  -Server: v1.14.0

kubeadm version : v1.14.1



alisd@kubeMaster:~$ kubectl get pods -n kube-system
NAME                                 READY   STATUS    RESTARTS   AGE
calico-node-944jv                    2/2     Running   4          45h
coredns-fb8b8dccf-hzzwf              1/1     Running   2          45h
coredns-fb8b8dccf-zb228              1/1     Running   2          45h
etcd-kubemaster                      1/1     Running   3          45h
kube-apiserver-kubemaster            1/1     Running   3          45h
kube-controller-manager-kubemaster   1/1     Running   3          45h
kube-proxy-l6wrc                     1/1     Running   3          45h
kube-scheduler-kubemaster            1/1     Running   3          45h
my-scheduler-66cf896bfb-8j8sr        1/1     Running   2          45h


alisd@kubeMaster:~$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
annotation-second-scheduler   0/1     Pending   0          4s



alisd@kubeMaster:~$ kubectl describe pod annotation-second-scheduler
Name:               annotation-second-scheduler
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             name=multischeduler-example
Annotations:        <none>
Status:             Pending
IP:                 
Containers:
  pod-with-second-annotation-container:
    Image:        k8s.gcr.io/pause:2.0
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jclk7 (ro)
Volumes:
  default-token-jclk7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jclk7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>







alisd@kubeMaster:~$ kubectl logs -f my-scheduler-66cf896bfb-8j8sr -n kube-system



E0426 14:44:01.742799       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0426 14:44:02.743952       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope

.....

alisd@kubeMaster:~$ kubectl get clusterrolebinding
NAME                                                   AGE
calico-node                                            46h
cluster-admin                                          46h
kubeadm:kubelet-bootstrap                              46h
kubeadm:node-autoapprove-bootstrap                     46h
kubeadm:node-autoapprove-certificate-rotation          46h
kubeadm:node-proxier                                   46h
my-scheduler-as-kube-scheduler                         46h

......

alisd@kubeMaster:~$ kubectl describe clusterrolebinding my-scheduler-as-kube-scheduler
Name:         my-scheduler-as-kube-scheduler
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  system:kube-scheduler
Subjects:
  Kind            Name          Namespace
  ----            ----          ---------
  ServiceAccount  my-scheduler  kube-system

........

alisd@kubeMaster:~$ kubectl describe serviceaccount my-scheduler -n kube-systemName:                my-scheduler
Namespace:           kube-system
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   my-scheduler-token-68pvk
Tokens:              my-scheduler-token-68pvk
Events:              <none>

.......

-- ali saaad
kubernetes
scheduler

Similar Questions

helm configmap include client script in values.yaml
Pods not connecting with service having same labels -Kubernetes
adding master to Kubernetes cluster: cluster doesn't have a stable controlPlaneEndpoint address
How to set a minimum scale for Cloud Run on GKE services?
How to set requests per second limit on GKE and Kong Ingress?

3 Answers

8/27/2019

To add to ali saaad solution answer I also had to add "csinodes" to resources to be able to schedule pod, so it looks like:

- apiGroups:
   - storage.k8s.io
   resources:
   - csinodes
   - storageclasses
   verbs:
   - watch
   - list
   - get
-- aurimas
Source: StackOverflow
4/26/2019

Service Account system:serviceaccount:kube-system:my-scheduler needs to associate with the following cluster wide role system:kube-scheduler in order to access the resources. my-scheduler will have same authorities as the default scheduler.

Note also that we created a dedicated service account my-scheduler and bind the cluster role system:kube-scheduler to it so that it can acquire the same privileges as kube-scheduler. define-a-kubernetes-deployment-for-the-scheduler

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
-- Suresh Vishnoi
Source: StackOverflow
4/29/2019

I've found a solution

Add these lines:

- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - watch
  - list
  - get

to the end of the output of this command (this opens a file for you to edit):

kubectl edit clusterrole system:kube-scheduler

The pod using the scheduler that I deployed is now Running

alisd@kubeMaster:~$ kubectl get pods 
NAME                          READY   STATUS    RESTARTS   AGE
annotation-second-scheduler   1/1     Running   0          9m33s

......

kubectl describe pod annotation-second-scheduler

......

 Events:
      Type    Reason     Age   From                 Message
      ----    ------     ----  ----                 -------
      Normal  Scheduled  12m   my-scheduler         Successfully assigned default/annotation-second-scheduler to kubemaster
      Normal  Pulled     12m   kubelet, kubemaster  Container image "k8s.gcr.io/pause:2.0" already present on machine
      Normal  Created    12m   kubelet, kubemaster  Created container pod-with-second-annotation-container
      Normal  Started    12m   kubelet, kubemaster  Started container pod-with-second-annotation-container
-- ali saaad
Source: StackOverflow