EKS - How to annotate some nodes in USERDATA?

7/11/2019

To prevent Cluster Auto Scaler from terminating some of nodes, I would need to annotate them with:

cluster-autoscaler.kubernetes.io/scale-down-disabled=true;

Is there a way to do so in USERDATA script? For labeling the nodes, there is no issue, and it is possible to do so via:

    --kubelet-extra-args \
        "--node-labels=

Thanks

-- imriss
annotations
eks
kubernetes
nodes

3 Answers

7/22/2019

No, it not possible.

The list of supported parameters for the bootstrap script:

--use-max-pods Sets --max-pods for the kubelet when true. (default: true)
--b64-cluster-ca The base64 encoded cluster CA content. Only valid when used with --apiserver-endpoint. Bypasses calling \"aws eks describe-cluster\"
--apiserver-endpoint The EKS cluster API Server endpoint. Only valid when used with --b64-cluster-ca. Bypasses calling \"aws eks describe-cluster\"
--kubelet-extra-args Extra arguments to add to the kubelet. Useful for adding labels or taints.
--enable-docker-bridge Restores the docker default bridge network. (default: false)
--aws-api-retry-attempts Number of retry attempts for AWS API call (DescribeCluster) (default: 3)
--docker-config-json The contents of the /etc/docker/daemon.json file. Useful if you want a custom config differing from the default one in the AMI
-- A_Suh
Source: StackOverflow

12/20/2019

It is absolutely possible. Here is part of my example userdata, specifically useful if you want to run both OnDemand and Spot instance. In my example I am adding lifecycle node label which changes based on the type. See below:

--use-max-pods 'true' \
--kubelet-extra-args ' --node-labels=lifecycle=OnDemand \
--system-reserved cpu=250m,memory=0.2Gi,ephemeral-storage=1Gi \
--kube-reserved cpu=250m,memory=1Gi,ephemeral-storage=1Gi \
--eviction-hard memory.available<0.2Gi,nodefs.available<10% \
--event-qps 0'

I hope that gives you a nice example.

-- marcincuber
Source: StackOverflow

12/20/2019

You can add node labels, taints, etc by using the --kubelet-extra-args option on the bootstrap.sh invokation as you guessed. For an example, see the AWS Blog post: Improvements for Amazon EKS Worker Node Provisioning

Use a USERDATA script similar to the following:

      UserData: !Base64
        "Fn::Sub": |
          #!/bin/bash
          set -o xtrace
          /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
          /opt/aws/bin/cfn-signal --exit-code $? \
                   --stack  ${AWS::StackName} \
                   --resource NodeGroup  \
                   --region ${AWS::Region}

The above is a fragment from the CloudFormation template. Of course you can make your script more complex, with security hardening, etc. if you so desire.

For a complete CloudFormation template, download the sample from AWS:

curl -O https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-11-15/amazon-eks-nodegroup.yaml
-- dlaidlaw
Source: StackOverflow