To prevent Cluster Auto Scaler
from terminating some of nodes, I would need to annotate them with:
cluster-autoscaler.kubernetes.io/scale-down-disabled=true;
Is there a way to do so in USERDATA
script? For labeling the nodes, there is no issue, and it is possible to do so via:
--kubelet-extra-args \
"--node-labels=
Thanks
No, it not possible.
The list of supported parameters for the bootstrap script:
--use-max-pods Sets --max-pods for the kubelet when true. (default: true)
--b64-cluster-ca The base64 encoded cluster CA content. Only valid when used with --apiserver-endpoint. Bypasses calling \"aws eks describe-cluster\"
--apiserver-endpoint The EKS cluster API Server endpoint. Only valid when used with --b64-cluster-ca. Bypasses calling \"aws eks describe-cluster\"
--kubelet-extra-args Extra arguments to add to the kubelet. Useful for adding labels or taints.
--enable-docker-bridge Restores the docker default bridge network. (default: false)
--aws-api-retry-attempts Number of retry attempts for AWS API call (DescribeCluster) (default: 3)
--docker-config-json The contents of the /etc/docker/daemon.json file. Useful if you want a custom config differing from the default one in the AMI
It is absolutely possible. Here is part of my example userdata, specifically useful if you want to run both OnDemand and Spot instance. In my example I am adding lifecycle node label which changes based on the type. See below:
--use-max-pods 'true' \
--kubelet-extra-args ' --node-labels=lifecycle=OnDemand \
--system-reserved cpu=250m,memory=0.2Gi,ephemeral-storage=1Gi \
--kube-reserved cpu=250m,memory=1Gi,ephemeral-storage=1Gi \
--eviction-hard memory.available<0.2Gi,nodefs.available<10% \
--event-qps 0'
I hope that gives you a nice example.
You can add node labels, taints, etc by using the --kubelet-extra-args
option on the bootstrap.sh invokation as you guessed. For an example, see the AWS Blog post: Improvements for Amazon EKS Worker Node Provisioning
Use a USERDATA script similar to the following:
UserData: !Base64
"Fn::Sub": |
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
/opt/aws/bin/cfn-signal --exit-code $? \
--stack ${AWS::StackName} \
--resource NodeGroup \
--region ${AWS::Region}
The above is a fragment from the CloudFormation template. Of course you can make your script more complex, with security hardening, etc. if you so desire.
For a complete CloudFormation template, download the sample from AWS:
curl -O https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-11-15/amazon-eks-nodegroup.yaml