I am running a 4 worker node cluster in GCP.
And the current status of my nodes is :
Node A
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system calico-node-bfpbd 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-7br2g 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-7kvcm 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
kube-system tiller-deploy-56c4cf647b-5vsvb 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 310m (7%) 620m (15%)
memory 174Mi (1%) 1006Mi (7%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:08 -0400 Thu, 01 Nov 2018 21:31:08 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:31:06 -0400 KubeletReady kubelet is posting ready statusNode B
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
jenkins-test jenkins-master 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system calico-node-qglbv 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-g74ff 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-bvczb 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 310m (7%) 620m (15%)
memory 174Mi (1%) 1006Mi (7%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:06 -0400 Thu, 01 Nov 2018 21:31:06 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:31:06 -0400 KubeletReady kubelet is posting ready statusNode C
Non-terminated Pods: (3 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system calico-node-w9px6 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-4r2ck 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-r92xs 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:01 -0400 Thu, 01 Nov 2018 21:31:01 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:31:09 -0400 KubeletReady kubelet is posting ready statusNode D
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system addons-kubernetes-dashboard-8656b6fc5f-68wzm 50m (1%) 200m (5%) 50Mi (0%) 256Mi (1%)
kube-system addons-nginx-ingress-controller-77579b6d64-sqzl7 100m (2%) 300m (7%) 100Mi (0%) 512Mi (3%)
kube-system addons-nginx-ingress-nginx-ingress-k8s-backend-5d6d4598ff-nfzt4 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system calico-node-h2t5b 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system coredns-5c554d9f6f-fnwqq 100m (2%) 200m (5%) 15Mi (0%) 80Mi (0%)
kube-system kube-proxy-bfhjr 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system metrics-server-7f4cbf557d-985sj 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system node-exporter-frdrd 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
kube-system vpn-shoot-7bcd5f4bb-88sc7 100m (2%) 300m (7%) 128Mi (0%) 512Mi (3%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:30:54 -0400 Thu, 01 Nov 2018 21:30:54 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:31:02 -0400 KubeletReady kubelet is posting ready statusAs all the results show, my nodes are all healthy and have ample resources with 4CPUs each and 16GB memory each.
Now when I try to deploy my second statefulSet in my namespaces, the Pod remains in Pending state. The describe shows the below message:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m (x123 over 7m) default-scheduler pod has unbound PersistentVolumeClaims (repeated 2 times)
Normal NotTriggerScaleUp 12s (x26 over 6m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added)I also drained 2 of the nodes, labeled the two nodes and attached a nodeSelector to my statefulset to only deploy on those 2 almost empty nodes but the result is the same.
I'm not sure why my pod is trying to scale-up. That's not the intention. Any help will be greatly appreciated.
kubectl get po -n jenkins-test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
jenkins-agent 0/1 Pending 0 12m <none> <none> <none>
jenkins-master 1/1 Running 0 22m 100.96.1.2 shoot--t--csp-worker-hqh6g-z1-6df8f7dc66-bcj6t <none>pod has unbound PersistentVolumeClaims , I think it's the key to this problem. About scale up, Pod is created through 'scale from 0 to any instance you defined'.