Update: found that instances has a limit to number of pods.
Does that mean that auto-scaling group won't increase the node count unless the CPU or memory resources hit that limit?
I'm running t3.medium instance and looking at the nodes describe command output it hasn't used 100% of resources. But I get the error 0/1 nodes are available: 1 Too many pods..
Another thing is I have auto scaling configured for cluster node group, and the max size is 2 but second node hasn't spawn yet which means there is still resources right?
scaling_config {
desired_size = 1
max_size = 2
min_size = 1
}
Can someone explain why it can't run new pod and what's blocking it?
kubectl get po -A:
NAMESPACE NAME READY STATUS RESTARTS AGE
default alertmanager-prometheus-prometheus-oper-alertmanager-0 2/2 Running 0 3h34m
default mongo-express-5b8b8989c8-b7qq8 1/1 Running 0 3h33m
default mongodb-0 1/1 Running 0 3h37m
default mongodb-1 1/1 Running 0 3h36m
default mongodb-arbiter-0 1/1 Running 0 3h36m
default mongodb-exporter-prometheus-mongodb-exporter-786999d67c-c586f 1/1 Running 0 123m
default nginx-ingress-controller-868b5465b4-8klbr 1/1 Running 0 3h45m
default nginx-ingress-default-backend-7c868597f4-n85s8 1/1 Running 0 3h45m
default prometheus-grafana-67596ff846-nkssr 2/2 Running 0 3h35m
default prometheus-kube-state-metrics-c65b87574-g6pt7 1/1 Running 0 3h35m
default prometheus-prometheus-node-exporter-kl5b2 1/1 Running 0 3h35m
default prometheus-prometheus-oper-operator-5ff8fbd5fb-fzgvf 2/2 Running 0 3h35m
default prometheus-prometheus-prometheus-oper-prometheus-0 3/3 Running 1 3h34m
kube-system aws-node-7t29d 1/1 Running 0 5h58m
kube-system coredns-6b4cdc67b4-7mnjh 1/1 Running 0 6h2m
kube-system coredns-6b4cdc67b4-7px74 1/1 Running 0 6h2m
kube-system kube-proxy-dl2dr 1/1 Running 0 5h58m
test-db-dev db01-0 0/1 Pending 0 11m
`kubectl describe node`:
Name: ip-10-0-0-250.ap-south-1.compute.internal
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.medium
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=test_nodes-dev
eks.amazonaws.com/nodegroup-image=ami-0ad87e82a7f1e7f4b
failure-domain.beta.kubernetes.io/region=ap-south-1
failure-domain.beta.kubernetes.io/zone=ap-south-1b
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-0-0-250.ap-south-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t3.medium
topology.kubernetes.io/region=ap-south-1
topology.kubernetes.io/zone=ap-south-1b
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 09 Dec 2020 03:43:22 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: ip-10-0-0-250.ap-south-1.compute.internal
AcquireTime: <unset>
RenewTime: Wed, 09 Dec 2020 09:39:11 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 09 Dec 2020 09:39:04 -0500 Wed, 09 Dec 2020 03:43:22 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 09 Dec 2020 09:39:04 -0500 Wed, 09 Dec 2020 03:43:22 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 09 Dec 2020 09:39:04 -0500 Wed, 09 Dec 2020 03:43:22 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 09 Dec 2020 09:39:04 -0500 Wed, 09 Dec 2020 03:43:42 -0500 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.0.0.250
Hostname: ip-10-0-0-250.ap-south-1.compute.internal
InternalDNS: ip-10-0-0-250.ap-south-1.compute.internal
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 20959212Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3977908Ki
pods: 17
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 1930m
ephemeral-storage: 18242267924
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3422900Ki
pods: 17
System Info:
Machine ID: ec2474d1824a7ec75cfb9c1704c92d34
System UUID: EC2474D1-824A-7EC7-5CFB-9C1704C92D34
Boot ID: 3ddcd32a-8b7a-4a21-b41b-37a51686ca67
Kernel Version: 4.14.203-156.332.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.6
Kubelet Version: v1.18.9-eks-d1db3c
Kube-Proxy Version: v1.18.9-eks-d1db3c
ProviderID: aws:///ap-south-1b/i-047e645215dc3c7e1
Non-terminated Pods: (17 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default alertmanager-prometheus-prometheus-oper-alertmanager-0 100m (5%) 100m (5%) 225Mi (6%) 25Mi (0%) 3h32m
default mongo-express-5b8b8989c8-b7qq8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h31m
default mongodb-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h34m
default mongodb-1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h34m
default mongodb-arbiter-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h34m
default mongodb-exporter-prometheus-mongodb-exporter-786999d67c-c586f 0 (0%) 0 (0%) 0 (0%) 0 (0%) 121m
default nginx-ingress-controller-868b5465b4-8klbr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h43m
default nginx-ingress-default-backend-7c868597f4-n85s8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h43m
default prometheus-grafana-67596ff846-nkssr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h32m
default prometheus-kube-state-metrics-c65b87574-g6pt7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h32m
default prometheus-prometheus-node-exporter-kl5b2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h32m
default prometheus-prometheus-oper-operator-5ff8fbd5fb-fzgvf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h32m
default prometheus-prometheus-prometheus-oper-prometheus-0 200m (10%) 200m (10%) 50Mi (1%) 50Mi (1%) 3h32m
kube-system aws-node-7t29d 10m (0%) 0 (0%) 0 (0%) 0 (0%) 5h55m
kube-system coredns-6b4cdc67b4-7mnjh 100m (5%) 0 (0%) 70Mi (2%) 170Mi (5%) 6h
kube-system coredns-6b4cdc67b4-7px74 100m (5%) 0 (0%) 70Mi (2%) 170Mi (5%) 6h
kube-system kube-proxy-dl2dr 100m (5%) 0 (0%) 0 (0%) 0 (0%) 5h55m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 610m (31%) 300m (15%)
memory 415Mi (12%) 415Mi (12%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>