Cannot create ACI for new AKS cluster connected to existing on-prem Vnet

2/25/2021

I'm attempting to create a new AKS cluster using Kubernetes version 1.19.7 and virtual machine scale sets and connect it to an existing on-prem vnet. On my first attempt, everything succeeded except for the creation of the actual ACI in Azure. The aci-connector node got created in Kubernetes but remained in a CrashLoopBackOff state, each time with the following error in the kubernetes logs:

Error: error initializing provider azure: error setting up network profile: unable to delegate subnet 'xxxxxxxxx' to Azure Container Instance since it references the route table '/subscriptions/yyyyyyyy/resourceGroups/zzzzzzzz/providers/Microsoft.Network/routeTables/rrrrrrr'.

I tried recreating the cluster differently, according to limitations buried in MS documentation (using service principal, with empty subnet containing no other resources, with proper role permissions applied to the service account). Still no luck. Tried a few other tweaks on the networking side as well, but to no avail.

Here are the Azure CLI commands I used (names obfuscated) with/without service principal:

Using managed identity

az aks create -g yyyyyyyyy -n zzzzzzzz --aad-admin-group-object-ids 00000000-0000-0000-0000-000000000000 --aci-subnet-name myAciSubnet --assign-identity /subscriptions/xxxxxxx/resourcegroups/yyyyyyy/providers/Microsoft.ManagedIdentity/userAssignedIdentities/k8s-admin-qa --docker-bridge-address 172.17.0.1/16 --dns-service-ip 10.2.0.10 --enable-aad --enable-addons virtual-node --enable-managed-identity --generate-ssh-keys --kubernetes-version 1.19.7 --location eastus2 --network-plugin azure --service-cidr 10.2.0.0/16 --subscription xxxxxxx --vnet-subnet-id /subscriptions/xxxxxxx/resourceGroups/myNetworkResourceGroup/providers/Microsoft.Network/virtualNetworks/myVnet/subnets/mySubnet

Using Service Principal

az aks create -g yyyyyyy -n zzzzzzz --aad-admin-group-object-ids 00000000-0000-0000-0000-000000000000 --aci-subnet-name myAciSubnet --docker-bridge-address 172.17.0.1/16 --dns-service-ip 10.2.0.10 --enable-aad --enable-addons virtual-node --generate-ssh-keys --kubernetes-version 1.19.7 --location eastus2 --network-plugin azure --service-cidr 10.2.0.0/16 --subscription xxxxxxx --vnet-subnet-id /subscriptions/xxxxxxx/resourceGroups/myNetworkResourceGroup/providers/Microsoft.Network/virtualNetworks/myVnet/subnets/mySubnet  --service-principal ppppppppp --client-secret SSSSSSSSSSS 

If anyone out there has been able to successfully deploy/configure an AKS cluster using ACI with virtual machine scale sets, connected to an on-prem network, or can otherwise assist in troubleshooting or configuration, I'd love to hear from you!

-- ChuckH
azure
kubernetes
networking

1 Answer

2/26/2021

The subnet for the ACI should be no other resources except the ACI and also no attached route table. Because Azure will attach a profile of the container group for it. And the error shows the subnet you want to use for the ACI already attached a routing table. So you can create a new subnet with nothing or just disassociated the routing table from the subnet.

-- Charles Xu
Source: StackOverflow