Okay, so I messed up, I accidentally ran az ad sp reset-credentials
against the Service Principal that our AKS cluster runs under. And now we are getting errors like:
Error creating load balancer (will retry): error getting LB for service test/admin-api: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/****/resourceGroups/MC_****/providers/Microsoft.Network/loadBalancers?api-version=2017-09-01: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided.\r\nTrace ID:****\r\nCorrelation ID:**** \r\nTimestamp: 2018-08-23 12:01:33Z","error_codes":[70002,50012],"timestamp":"2018-08-23 12:01:33Z","trace_id":"****","correlation_id":"****"}
and
Failed to pull image "****.azurecr.io/****:****": rpc error: code = Unknown desc = Error response from daemon: Get https://****.azurecr.io/v2/****/manifests/****: unauthorized: authentication required
So now I want to find the original client secret that the Service Principal uses, so that I can re-add that as a key to the Service Principal. That's the only solution I can think of other than recreating the entire cluster.
Any ideas?
It's an annoying thing that you want to do. For your issue, you cannot pull the image without authentication.
First, you have to find out the service principal of your container registry. You can do this in the Azure portal and navigate to the registry panel, then you can find the service principal like this:
Or you can use the Azure CLI command to find the registry ID like this:
az acr show --resource-group groupName --name registryName --query id --output tsv
Then use the command to find the service principal ID like this:
az role assignment list --scope registryID
You can select the service principal which you want.
Then you would get all the secrets with the command kubectl get secrets
and kubectl get secrets secretName -o yaml
to get the token of the secret. Then analyze one-by-one to check if the username the same as the service principal ID. You can use tools such as JWT to analyze the secret token. The result will like this:
If the username the same as the service principal ID which you find, that is the secret you want. This step is a trouble. You should check the secret one-by-one or you will have a more great way to check them.
By the way, it seems that you can just see the password of the service principal once when you create it. The Azure will not show you again. But if you create the Kubernetes secret, the password is stored in it.
In the end the solution was quite simple.
MC_<resourcegroup>_<aksName>_<region>
.cat /etc/kubernetes/azure.json
and click "Run"The command will return the contents of the JSON file. The property you need is aadClientSecret
Whoever comes over this issue there's an updated solution from Microsoft
They also mention (something that's not obvious) that: By default, AKS clusters are created with a service principal that has a one-year expiration time.
Also, As of Azure CLI 2.0.68, the --password parameter to create a service principal with a user-defined password is no longer supported to prevent the accidental use of weak passwords. so the initial solution to change the service principal password doesn't work anymore.