K
Q

How can I ensure I annotate each Kubernetes node before Pods are scheduled to it on EKS?

3/27/2019

I am trying to use CNI Custom Networking on EKS to make sure that Pod IPs are allocated from alternative subsets (to prevent IP starvation in the subnets my cluster nodes are running in). To do this I need to create some ENIConfigs and annotate each node.

How can I ensure that each node is annotated before any Pods are scheduled to it to ensure no Pod IPs are allocated from the subnets my nodes are running in?

EDIT: The only solution I can think of so far is:

  • Add a NoSchedule taint to all nodes by default
  • Deploy a custom controller that tolerates the taint
  • Get the controller to annotate all nodes as required and remove the taint

However, if the above is the only workaround that is a lot of effort for a managed service

-- dippynark
amazon-eks
aws-eks
cni
kubernetes

Similar Questions

Exposing Kafka cluster in Kubernetes using LoadBalancer service
Minikube: Path to Root Certificate
equivalent of "minikube ssh" with docker-for-desktop Kubernetes node
Kubernetes config file from host to container?

1 Answer

3/27/2019

How about:

  • Add a ENIConfigComplete: false taint to all nodes by default
  • Deploy DaemonSet that tolerates ENIConfigComplete: false
  • DaemonSet creates a pod on each new node which
  • creates some ENIConfigs on the node (bash script??)
  • annotates each node with ENIConfigComplete: true
  • DaemonSet no longer tolerates the node, so
  • Pod is removed from the node.

The DaemonSet would ensure that every new node was properly set up.

Salesforce talk about this technique for provisioning the disks on their new nodes:

It would avoid having a long running controller process.

-- richardw
Source: StackOverflow