Kubernetes exhausting IPs on a single subnet

12/20/2021

My current eks cluster is assoicated with 4 subnets, out of 4 its exhausting the ip allocations from the a single subnet meaning no IPs left to allocate to pods. All of the available subnets have a CIDR block of /24. Can anyone help me with to understand how the allocations work and what could be the possible reason it's considering only a certain Subnet group and not having a distributed allocations on all subnets?

Thanks in advance.

-- gkcld
amazon-eks
amazon-vpc
amazon-web-services
kubernetes

1 Answer

12/20/2021

Do you have workers on multiple subnets? If you don't, then only your nodes in a single subnet will have pods added to them.

A good practice for EKS is to create an ASG for workers per subnet, this ensures that you always have nodes in that subnets as long as that zone is up and running.

Moreover, this will solve the problem of nodes with a persistent volume respawning on the wrong subnet and there for not being able to connect to the volume.

This is how I build out my subnets via terraform.

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name                 = local.cluster_name
  cidr                 = module.subnet_addrs.base_cidr_block
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = local.private_subnets
  public_subnets       = [module.subnet_addrs.network_cidr_blocks.pub1, module.subnet_addrs.network_cidr_blocks.pub2, module.subnet_addrs.network_cidr_blocks.pub3]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  reuse_nat_ips        = true # <= Skip creation of EIPs for the NAT Gateways
  external_nat_ip_ids  = aws_eip.nat.*.id

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
  tags = local.tags
}

module "eks" {
  source = "terraform-aws-modules/eks/aws"

  cluster_name    = local.cluster_name
  cluster_version = var.cluster_version
  subnets         = module.vpc.private_subnets
  vpc_id          = module.vpc.vpc_id
  enable_irsa     = true
  map_roles       = concat(var.map_roles, local.map_roles)
  map_users       = var.map_users
  cluster_encryption_config = [
    {
      provider_key_arn = aws_kms_key.eks.arn
      resources        = ["secrets"]
    }
  ]
  cluster_endpoint_public_access_cidrs = var.cluster_endpoint_public_access_cidrs
  tags                                 = local.tags
  worker_groups                        = var.worker_groups

  worker_groups_launch_template = [
    {
      name                 = "bottlerocket-nodes"
      ami_id               = local.bottlerocket_ami
      instance_type        = "t3a.small"
      asg_desired_capacity = 1

      # This section overrides default userdata template to pass bottlerocket
      # specific user data
      userdata_template_file = "${path.module}/userdata.toml"
      # we are using this section to pass additional arguments for
      # userdata template rendering
      userdata_template_extra_args = {
        enable_admin_container   = false
        enable_control_container = true
        aws_region               = var.aws_region
      }
      # example of k8s/kubelet configuration via additional_userdata
      additional_userdata = <<EOT
[settings.kubernetes.node-labels]
ingress = "allowed"
EOT
    }
  ]
  node_groups_defaults = {
    ami_type = "AL2_x86_64"
  }

  node_groups = local.node_groups
}
-- Josh Beauregard
Source: StackOverflow