When I trying to follow the Kubernetes Configure Multiple Schedulers guide, I cannot successfully get the my-scheduler pod running.

I succeed to build the image and push it to a registry, and the image shows in my docker and Google Could Platform. But when I apply the deployment file to set the pods, the status is CrashLoopBackOff.

my-scheduler-64576b7897-8b549            0/1     CrashLoopBackOff   14         84m

Here is the output for kuberctl describe pod my-scheduler-64576b7897-8b549 -n kube-system:

Name:               my-scheduler-64576b7897-8b549
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Tue, 07 May 2019 15:06:59 -0500
Labels:             component=scheduler
                    pod-template-hash=64576b7897
                    tier=control-plane
                    version=second
Annotations:        <none>
Status:             Running
IP:                 172.17.0.15
Controlled By:      ReplicaSet/my-scheduler-64576b7897
Containers:
  kube-second-scheduler:
    Container ID:  docker://a0cadf03dd5ffd11cbb6f1f8e669fc04cd6aa00aa6f361550f06f32785a4d63c
    Image:         gcr.io/my-gcp-project-239900/my-kube-scheduler:1.0
    Image ID:      docker-pullable://gcr.io/my-gcp-project-239900/my-kube-scheduler@sha256:0895687b7646d159438f086e8f8b23b253332930e169cf29d8b9951ebf3ba653
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/kube-scheduler
      --address=1.1.1.1
      --leader-elect=false
      --scheduler-name=my-scheduler
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 07 May 2019 15:07:45 -0500
      Finished:     Tue, 07 May 2019 15:07:45 -0500
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 07 May 2019 15:07:24 -0500
      Finished:     Tue, 07 May 2019 15:07:24 -0500
    Ready:          False
    Restart Count:  3
    Requests:
      cpu:        100m
    Liveness:     http-get http://:10251/healthz delay=15s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:10251/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from my-scheduler-token-lzwvk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  my-scheduler-token-lzwvk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-scheduler-token-lzwvk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  48s               default-scheduler  Successfully assigned kube-system/my-scheduler-64576b7897-8b549 to minikube
  Normal   Pulled     2s (x4 over 47s)  kubelet, minikube  Container image "gcr.io/my-gcp-project-239900/my-kube-scheduler:1.0" already present on machine
  Normal   Created    2s (x4 over 47s)  kubelet, minikube  Created container kube-second-scheduler
  Normal   Started    2s (x4 over 47s)  kubelet, minikube  Started container kube-second-scheduler
  Warning  BackOff    1s (x6 over 45s)  kubelet, minikube  Back-off restarting failed container

And when I try to see the log of this pod, this error message appears:

 standard_init_linux.go:207: exec user process caused "exec format error"

And my Dockerfile is bit different from the guide, it looks like:

FROM busybox
ADD ./_output/local/bin/darwin/amd64/kube-scheduler /usr/local/bin/kube-scheduler

I changed

ADD ./_output/dockerized/bin/linux/amd64/kube-scheduler /usr/local/bin/kube-scheduler

into

ADD ./_output/local/bin/darwin/amd64/kube-scheduler /usr/local/bin/kube-scheduler

because ./_output/local/bin/darwin/amd64/kube-scheduleris where the scheduler file acctually located. I would also appreciate it if anyone could tell me why would this happen.

Some Updates

Different file location

Following the guide, I typed these command in my shell

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
make

by executing make, the Makefilefile will generate the kubernetes exec files in certain path (including the kube-scheduler we want). And the generation path is related to your system. Since I'm using minikube to run kubernetes and docker, and it runs on darwin(amd64). I think that's the reason why my path is different from the guide.

The "exec format error"

For this error

 standard_init_linux.go:207: exec user process caused "exec format error"

I find this issue suggesting add the #!/bin/bash before you try to run commands in the container. This tells the system to treat the commands as bash commands. Here in busybox it should be /bin/sh. (I don't know much about shell language mechanism, so correct me if I'm wrong)

So I changed these lines in my-sheduler.yaml:

- command:
 - /usr/local/bin/kube-scheduler
 - --address=0.0.0.0
 - --leader-elect=false
 - --scheduler-name=my-scheduler

into

- command:
 - /bin/sh
 - /usr/local/bin/kube-scheduler
 - --address=0.0.0.0
 - --leader-elect=false
 - --scheduler-name=my-schedule

Now the my-sheduler pods still fail and get the CrashLoopBackOff error. Good news is I now have the log when executing kubectl logs. Bad news is the logs are unreadable like this

/usr/local/bin/kube-scheduler: line 1: ����: not found
/usr/local/bin/kube-scheduler: line 2: syntax error: unexpected ")"

I assume this happened because of something happened in the docker image. But since I'm using the scrips right from the kubernetes github page, there shouldn't be any bug.

The same log shows up when I run docker run -it {my-image-id} /bin/sh and go into my image and run the exec kube-scheduler file.

I can't figure it out what's in the GO script causes this problem.

Another alternative

I ended up trying another way to build my custom scheduler.

I followed this tutorial: https://www.youtube.com/watch?v=IYcL0Un1io0&t=1807s

And you can find the source code here: https://github.com/kelseyhightower/scheduler

Comparing to the official guide, this one is more clear light (in in official guide you have to build the whole thing to get the scheduler file).

Hope this helps if someone is running into the same problem.