My current eks cluster is assoicated with 4 subnets, out of 4 its exhausting the ip allocations from the a single subnet meaning no IPs left to allocate to pods. All of the available subnets have a CIDR block of /24. Can anyone help me with to understand how the allocations work and what could be the possible reason it's considering only a certain Subnet group and not having a distributed allocations on all subnets?
Thanks in advance.
Do you have workers on multiple subnets? If you don't, then only your nodes in a single subnet will have pods added to them.
A good practice for EKS is to create an ASG for workers per subnet, this ensures that you always have nodes in that subnets as long as that zone is up and running.
Moreover, this will solve the problem of nodes with a persistent volume respawning on the wrong subnet and there for not being able to connect to the volume.
This is how I build out my subnets via terraform.
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
name = local.cluster_name
cidr = module.subnet_addrs.base_cidr_block
azs = data.aws_availability_zones.available.names
private_subnets = local.private_subnets
public_subnets = [module.subnet_addrs.network_cidr_blocks.pub1, module.subnet_addrs.network_cidr_blocks.pub2, module.subnet_addrs.network_cidr_blocks.pub3]
enable_nat_gateway = true
single_nat_gateway = true
enable_dns_hostnames = true
reuse_nat_ips = true # <= Skip creation of EIPs for the NAT Gateways
external_nat_ip_ids = aws_eip.nat.*.id
public_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
"kubernetes.io/role/elb" = "1"
}
private_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
"kubernetes.io/role/internal-elb" = "1"
}
tags = local.tags
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = local.cluster_name
cluster_version = var.cluster_version
subnets = module.vpc.private_subnets
vpc_id = module.vpc.vpc_id
enable_irsa = true
map_roles = concat(var.map_roles, local.map_roles)
map_users = var.map_users
cluster_encryption_config = [
{
provider_key_arn = aws_kms_key.eks.arn
resources = ["secrets"]
}
]
cluster_endpoint_public_access_cidrs = var.cluster_endpoint_public_access_cidrs
tags = local.tags
worker_groups = var.worker_groups
worker_groups_launch_template = [
{
name = "bottlerocket-nodes"
ami_id = local.bottlerocket_ami
instance_type = "t3a.small"
asg_desired_capacity = 1
# This section overrides default userdata template to pass bottlerocket
# specific user data
userdata_template_file = "${path.module}/userdata.toml"
# we are using this section to pass additional arguments for
# userdata template rendering
userdata_template_extra_args = {
enable_admin_container = false
enable_control_container = true
aws_region = var.aws_region
}
# example of k8s/kubelet configuration via additional_userdata
additional_userdata = <<EOT
[settings.kubernetes.node-labels]
ingress = "allowed"
EOT
}
]
node_groups_defaults = {
ami_type = "AL2_x86_64"
}
node_groups = local.node_groups
}