Correct way of using eksctl ClusterConfig with vpc-cni addon and pass maxPodsPerNode to launch template?

12/2/2021

I've been trying to create an EKS cluster with vpc-cni addon due to the pod restrictions for m5.xlarge VMs (57). After creation I can see it is passed to the launchtemplate object but when doing a node describe it still can allocate the previous (wrong?) number

ClusterConfig:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: exchange-develop
  region: us-east-1
  version: '1.21'

managedNodeGroups:
  - name: default
    labels:  
      worker: default
    instanceType: m5.xlarge
    desiredCapacity: 2
    minSize: 2
    maxSize: 4
    tags:
      'k8s.io/cluster-autoscaler/enabled': 'true'
      'k8s.io/cluster-autoscaler/exchange-develop': 'owned'
    iam: 
     attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::658464581062:policy/eks-csi-driver-policy
        - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
        - arn:aws:iam::658464581062:policy/ALBIngressControllerIAMPolicy
        - arn:aws:iam::658464581062:policy/ExternalDNSPlicy
        - arn:aws:iam::658464581062:policy/eks-cluster-autoscaler
    maxPodsPerNode: 110

availabilityZones: ['us-east-1c', 'us-east-1d']

iam:
  withOIDC: true

vpc:
  cidr: 10.10.0.0/16
  #autoAllocateIPv6: true
  # disable public access to endpoint and only allow private access
  clusterEndpoints:
    publicAccess: true
    privateAccess: true

addons:
- name: vpc-cni
  version: '1.10.1'

Launch template with redacted data:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=***

--
Content-Type: text/x-shellscript
Content-Type: charset="us-ascii"

#!/bin/sh
set -ex
sed -i -E "s/^USE_MAX_PODS=\"\\$\{USE_MAX_PODS:-true}\"/USE_MAX_PODS=false/" /etc/eks/bootstrap.sh
KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq ".maxPods=110" $KUBELET_CONFIG)" > $KUBELET_CONFIG
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=<>
API_SERVER_URL=<>
K8S_CLUSTER_DNS_IP=<>
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=exchange-develop,alpha.eksctl.io/nodegroup-name=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=default,worker=default,eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d --max-pods=58' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false

Node description:

Name:               ip-10-10-19-34.ec2.internal
Roles:              <none>
Labels:             alpha.eksctl.io/cluster-name=exchange-develop
                    alpha.eksctl.io/nodegroup-name=default
                    beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m5.xlarge
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/capacityType=ON_DEMAND
                    eks.amazonaws.com/nodegroup=default
                    eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd
                    eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d
                    eks.amazonaws.com/sourceLaunchTemplateVersion=1
                    failure-domain.beta.kubernetes.io/region=us-east-1
                    failure-domain.beta.kubernetes.io/zone=us-east-1c
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=<<
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=m5.xlarge
                    topology.kubernetes.io/region=us-east-1
                    topology.kubernetes.io/zone=us-east-1c
                    worker=default
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 02 Dec 2021 10:22:20 -0300
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 02 Dec 2021 11:18:31 -0300   Thu, 02 Dec 2021 10:22:18 -0300   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 02 Dec 2021 11:18:31 -0300   Thu, 02 Dec 2021 10:22:18 -0300   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 02 Dec 2021 11:18:31 -0300   Thu, 02 Dec 2021 10:22:18 -0300   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 02 Dec 2021 11:18:31 -0300   Thu, 02 Dec 2021 10:22:40 -0300   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   10.10.19.34
  ExternalIP:   <<
  Hostname:     <<
  InternalDNS:  <<
  ExternalDNS:  <<
Capacity:
 attachable-volumes-aws-ebs:  25
 cpu:                         4
 ephemeral-storage:           83873772Ki
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      15921236Ki
 pods:                        58
Allocatable:
 attachable-volumes-aws-ebs:  25
 cpu:                         3920m
 ephemeral-storage:           76224326324
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      14904404Ki
 pods:                        58
System Info:
 Machine ID:                 ec28ac2717ec395cdf5b4e37f7672569
 System UUID:                ec28ac27-17ec-395c-df5b-4e37f7672569
 Boot ID:                    50b3d3d9-5dfa-40b6-99c8-20873632c7ca
 Kernel Version:             5.4.156-83.273.amzn2.x86_64
 OS Image:                   Amazon Linux 2
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://20.10.7
 Kubelet Version:            v1.21.5-eks-bc4871b
 Kube-Proxy Version:         v1.21.5-eks-bc4871b
ProviderID:                  aws:///<<<
Non-terminated Pods:         (2 in total)
  Namespace                  Name                CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                ------------  ----------  ---------------  -------------  ---
  kube-system                aws-node-9z7pw      25m (0%)      0 (0%)      0 (0%)           0 (0%)         61m
  kube-system                kube-proxy-2slc8    100m (2%)     0 (0%)      0 (0%)           0 (0%)         61m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests   Limits
  --------                    --------   ------
  cpu                         125m (3%)  0 (0%)
  memory                      0 (0%)     0 (0%)
  ephemeral-storage           0 (0%)     0 (0%)
  attachable-volumes-aws-ebs  0          0
Events:                       <none>

See allocatable pods 58...

So, what is the correct way of using eksctl to create a cluster with vpc-cni and pass the maxPodsPerNode argument to ec2 launch template?

EDIT:

Other things I've tried:

  • Create cluster from scratch with vpc-cni addon 2 managedNodeGroups with maxPodsPerNode and without, both will take the value of 58 and not 110

  • Add another nodegroup with eksctl create nodegroup, still 58

  • Add another nodegroup with EKS AWS UI, still 58

Eksctl version 0.75.0 Kubectl version 1.21.2

-- paltaa
amazon-eks
amazon-vpc
amazon-web-services
eksctl
kubernetes

1 Answer

12/3/2021

For managedNodeGroup you need to specify the AMI ID:

aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id --region us-east-1 --query "Parameter.Value" --output text

managedNodeGroups:
  - name: default
    ...
    maxPodsPerNode: 110
    ami: ami-00836a7940260f6dd
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd'
-- gohm&#39;c
Source: StackOverflow