I've run into a problem accessing Azure Key Vault whilst converting some services from Microsoft Service Fabric to Kubernetes. Within our ASP.NET core ConfigureServices call, we call AddAzureKeyVault from Microsoft.Extensions.Configuration.AzureKeyVaultConfigurationExtensions to inject some sensitive parts of our configuration as shown below.

public void ConfigureServices(IServiceCollection services)
{

    var config = new ConfigurationBuilder()
        .SetBasePath(AppContext.BaseDirectory)
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddAzureKeyVault(
            Environment.GetEnvironmentVariable("KEYVAULT_LOCATION"),
            Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_ID"),
            Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_SECRET")
        ).Build();

        //...
}

This works fine in a docker image running locally, yet once deployed into Azure Kubernetes service the pod is failing with the following...

Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'BadRequest'
   at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)

Looking at the source of Microsoft.Extensions.Configuration.AzureKeyVaultConfigurationExtensions I can reproduce the issue with this minimal code.

public void ConfigureServices(IServiceCollection services)
{
    GetSecrets(
        Environment.GetEnvironmentVariable("KEYVAULT_LOCATION"),
        Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_ID"),
        Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_SECRET")
    ).GetAwaiter().GetResult();
}

public async Task GetSecrets(string loc, string clientId, string clientSecret)
{
    KeyVaultClient.AuthenticationCallback callback =
        (authority, resource, scope) => GetTokenFromClientSecret(authority, resource, clientId, clientSecret);

    IKeyVaultClient _client = new KeyVaultClient(callback);

    //Exception thrown here
    var secrets = await _client.GetSecretsAsync(loc).ConfigureAwait(false);
}

private static async Task<string> GetTokenFromClientSecret(string authority, string resource, string clientId, string clientSecret)
{
    var authContext = new AuthenticationContext(authority);
    var clientCred = new ClientCredential(clientId, clientSecret);
    var result = await authContext.AcquireTokenAsync(resource, clientCred);
    return result.AccessToken;
}

My question is, what is different about this authentication when called from within a pod in AKS, as opposed to a local docker image, that would lead this call to fail?

I've confirmed the pod can access the wider internet, the key vault is not firewalled and Insights is showing some Authentication failures in the key vault logs. The Client Id and Secret are correct and have the correct rights since this works locally in Docker. What am I missing?