Pod not being scheduled even though resources available

11/2/2018

I am running a 4 worker node cluster in GCP.

And the current status of my nodes is :

Node A

Non-terminated Pods:         (4 in total)
  Namespace                  Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                              ------------  ----------  ---------------  -------------
  kube-system                calico-node-bfpbd                 250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                kube-proxy-7br2g                  50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                node-exporter-7kvcm               10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
  kube-system                tiller-deploy-56c4cf647b-5vsvb    0 (0%)        0 (0%)      0 (0%)           0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests    Limits
  --------  --------    ------
  cpu       310m (7%)   620m (15%)
  memory    174Mi (1%)  1006Mi (7%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:31:08 -0400   Thu, 01 Nov 2018 21:31:08 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:31:06 -0400   KubeletReady                 kubelet is posting ready status

Node B

Non-terminated Pods:         (4 in total)
  Namespace                  Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                  ------------  ----------  ---------------  -------------
  jenkins-test               jenkins-master    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                calico-node-qglbv                     250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                kube-proxy-g74ff                      50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                node-exporter-bvczb                   10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests    Limits
  --------  --------    ------
  cpu       310m (7%)   620m (15%)
  memory    174Mi (1%)  1006Mi (7%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:31:06 -0400   Thu, 01 Nov 2018 21:31:06 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:31:06 -0400   KubeletReady                 kubelet is posting ready status

Node C

Non-terminated Pods:         (3 in total)
  Namespace                  Name                   CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                   ------------  ----------  ---------------  -------------
  kube-system                calico-node-w9px6      250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                kube-proxy-4r2ck       50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                node-exporter-r92xs    10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:31:01 -0400   Thu, 01 Nov 2018 21:31:01 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:31:09 -0400   KubeletReady                 kubelet is posting ready status

Node D

Non-terminated Pods:         (9 in total)
  Namespace                  Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                                               ------------  ----------  ---------------  -------------
  kube-system                addons-kubernetes-dashboard-8656b6fc5f-68wzm                       50m (1%)      200m (5%)   50Mi (0%)        256Mi (1%)
  kube-system                addons-nginx-ingress-controller-77579b6d64-sqzl7                   100m (2%)     300m (7%)   100Mi (0%)       512Mi (3%)
  kube-system                addons-nginx-ingress-nginx-ingress-k8s-backend-5d6d4598ff-nfzt4    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                calico-node-h2t5b                                                  250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                coredns-5c554d9f6f-fnwqq                                           100m (2%)     200m (5%)   15Mi (0%)        80Mi (0%)
  kube-system                kube-proxy-bfhjr                                                   50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                metrics-server-7f4cbf557d-985sj                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                node-exporter-frdrd                                                10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
  kube-system                vpn-shoot-7bcd5f4bb-88sc7                                          100m (2%)     300m (7%)   128Mi (0%)       512Mi (3%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:30:54 -0400   Thu, 01 Nov 2018 21:30:54 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:31:02 -0400   KubeletReady                 kubelet is posting ready status

As all the results show, my nodes are all healthy and have ample resources with 4CPUs each and 16GB memory each.

Now when I try to deploy my second statefulSet in my namespaces, the Pod remains in Pending state. The describe shows the below message:

Events:
  Type     Reason             Age                From                Message
  ----     ------             ----               ----                -------
  Warning  FailedScheduling   2m (x123 over 7m)  default-scheduler   pod has unbound PersistentVolumeClaims (repeated 2 times)
  Normal   NotTriggerScaleUp  12s (x26 over 6m)  cluster-autoscaler  pod didn't trigger scale-up (it wouldn't fit if a new node is added)

I also drained 2 of the nodes, labeled the two nodes and attached a nodeSelector to my statefulset to only deploy on those 2 almost empty nodes but the result is the same.

I'm not sure why my pod is trying to scale-up. That's not the intention. Any help will be greatly appreciated.

kubectl get po -n jenkins-test -o wide
NAME                                      READY     STATUS    RESTARTS   AGE       IP           NODE                                             NOMINATED NODE
jenkins-agent         0/1       Pending   0          12m       <none>       <none>                                           <none>
jenkins-master        1/1       Running   0          22m       100.96.1.2   shoot--t--csp-worker-hqh6g-z1-6df8f7dc66-bcj6t   <none>
--
google-cloud-platform
kubernetes
kubernetes-statefulset

1 Answer

11/2/2018

pod has unbound PersistentVolumeClaims , I think it's the key to this problem. About scale up, Pod is created through 'scale from 0 to any instance you defined'.

-- Kun Li
Source: StackOverflow