I created an aks behind a firewall following the documentation from: https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#restrict-egress-traffic-using-azure-firewall

The only difference is that I create a cluster with a confidential computation VM with size Standard_DC2s_v2. The error is when Verifying sgx-plugins pods

kubectl get pods --all-namespaces

kube-system sgx-plugin-r6mmr 0/1 Init:CrashLoopBackOff 5 4m54s
kube-system sgx-webhook-787f99b5dc-qh2d5 0/1 Init:CrashLoopBackOff 4 5m50

How to reproduce it (as minimally and precisely as possible):

Define a set of environment variables to be used in resource creations.

Create Resource Group

az group create --name $RG --location $LOC

Create a virtual network with two subnets to host the AKS cluster and the Azure Firewall. Each will have their own subnet. Let's start with the AKS network.

az network vnet create
--resource-group $RG
--name $VNET_NAME
--location $LOC
--address-prefixes 10.42.0.0/16
--subnet-name $AKSSUBNET_NAME
--subnet-prefix 10.42.1.0/24

Dedicated subnet for Azure Firewall (Firewall name cannot be changed)

az network vnet subnet create
--resource-group $RG
--vnet-name $VNET_NAME
--name $FWSUBNET_NAME
--address-prefix 10.42.2.0/24

Create and set up an Azure Firewall with a UDR Create three standard SKU public IP resource that will be used as the Azure Firewall frontend address.

az network public-ip create -g $RG -n $FWPUBLICIP_NAME -l $LOC --sku "Standard"

Install Azure Firewall preview CLI extension

az extension add --name azure-firewall

Deploy Azure Firewall

az network firewall create -g $RG -n $FWNAME -l $LOC --enable-dns-proxy true

Configure Firewall IP Config

az network firewall ip-config create -g $RG -f $FWNAME -n $FWIPCONFIG_NAME --public-        ip-address $FWPUBLICIP_NAME --vnet-name $VNET_NAME

Capture Firewall IP Address for Later Use

FWPRIVATE_IP=$(az network firewall show -g $RG -n $FWNAME --query "ipConfigurations[0].privateIpAddress" -o tsv)
FWPUBLIC_IP_ID=$(az network firewall show -g $RG -n $FWNAME --query "ipConfigurations[0].publicIpAddress.id" -o tsv)
FWPUBLIC_IP=$(az network public-ip show --ids $FWPUBLIC_IP_ID --query "ipAddress" -o tsv)

or

FWPUBLIC_IP=$(az network public-ip show -g $RG -n $FWPUBLICIP_NAME --query "ipAddress" -o tsv)

Create UDR and add a route for Azure Firewall

az network route-table create -g $RG -l $LOC --name $FWROUTE_TABLE_NAME
az network route-table route create -g $RG --name $FWROUTE_NAME --route-table-name $FWROUTE_TABLE_NAME
--address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP
az network route-table route create -g $RG --name $FWROUTE_NAME_INTERNET --route-table-name $FWROUTE_TABLE_NAME
--address-prefix $FWPUBLIC_IP/32 --next-hop-type Internet

Add FW Network Rules needed for aks

az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'apiudp'
--protocols 'UDP' --source-addresses '' --destination-addresses "AzureCloud.$LOC"
--destination-ports 1194 --action allow --priority 100
az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'apitcp'
--protocols 'TCP' --source-addresses '' --destination-addresses "AzureCloud.$LOC"
--destination-ports 9000

az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'time'
--protocols 'UDP' --source-addresses '*' --destination-fqdns 'ntp.ubuntu.com' --destination-ports 123

az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'dnsserver'
--protocols 'UDP' --source-addresses '*' --destination-fqdns 10.41.0.10 --destination-ports 53

Add FW Application Rules

az network firewall application-rule create -g $RG -f $FWNAME --collection-name 'aksfwar' -n 'fqdn'
--source-addresses '*' --protocols 'http=80' 'https=443' --fqdn-tags "AzureKubernetesService"
--action allow --priority 100

Inspect the final firewall

az network firewall show -g $RG -n $FWNAME

Associate the route table to AKS Associate route table with next hop to Firewall to the AKS subnet

az network vnet subnet update -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --route-table $FWROUTE_TABLE_NAME

Deploy AKS with outbound type of UDR to the existing network Create SP and Assign Permission to Virtual Network

az ad sp create-for-rbac -n "${PREFIX}sp" --skip-assignment

it will return an exit like: { "appId": "xxxxx", "displayName": "xxx", "name": "xxx", "password": "xxxx", "tenant": "xxx" }

Now replace the APPID and PASSWORD below with the service principal appid and service principal password autogenerated by the previous command output. We'll reference the VNET resource ID to grant the permissions to the service principal so AKS can deploy resources into it. Assign SP Permission to VNET

az role assignment create --assignee $APPID --scope $VNETID --role "Network Contributor"

Deploy AKS Set the value for the subnet ID

SUBNETID=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME --name     $AKSSUBNET_NAME --query id -o tsv)

Create aks confidential computing cluster

az aks create -g $RG -n $AKSNAME -l $LOC
--node-count 1 --generate-ssh-keys
--network-plugin $PLUGIN
--outbound-type userDefinedRouting
--service-cidr 10.41.0.0/16
--dns-service-ip 10.41.0.10
--docker-bridge-address 172.17.0.1/16
--vnet-subnet-id $SUBNETID
--service-principal $APPID
--client-secret $PASSWORD
--api-server-authorized-ip-ranges $FWPUBLIC_IP/32
--enable-addon confcom

Add pool node

az aks nodepool add --cluster-name $AKSNAME --resource-group $RG --name confcompool1 --node-vm-size Standard_DC2s_v2 --node-count 1

Check the node list

az aks nodepool list --cluster-name $AKSNAME --resource-group $RG

Retrieve your IP address

use any that works

CURRENT_IP=$(dig @resolver1.opendns.com ANY myip.opendns.com +short)
CURRENT_IP=$(curl checkip.amazonaws.com)
CURRENT_IP=$(curl ifconfig.me)
CURRENT_IP=$(curl icanhazip.com)
CURRENT_IP=$(curl ipecho.net/plain)
CURRENT_IP=$(curl ifconfig.co)

Add to AKS approved list (better add it on the azure console for not to remove others access)

az aks update -g $RG -n $AKSNAME --api-server-authorized-ip-ranges $FWPUBLIC_IP/32,$CURRENT_IP/32

Use the az aks get-credentials command to configure kubectl to connect to your newly created Kubernetes cluster.

az aks get-credentials -g $RG -n $AKSNAME

Verify nodes

kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-confcompool1-15977364-vmss000000 Ready agent 6m37s v1.19.11
aks-nodepool1-15977364-vmss000000 Ready agent 18m v1.19.11

Verify sgx-plugins pods

kubectl get pods --all-namespaces

kube-system sgx-plugin-8n6bj 0/1 Init:CrashLoopBackOff 5 5m42s
kube-system sgx-webhook-787f99b5dc-mwfps 0/1 Init:CrashLoopBackOff 7 18m