I am deploying in Azure AKS a regular deployment and i want to use keyvault to store my secrets to get access to a database.

This is my deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    matchLabels:
      app: sonarqube
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      containers:
        - name: sonarqube
          image: sonarqube:8.9-developer
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 4096Mi
          volumeMounts:
          - mountPath: "/mnt/secrets/"
            name: secrets-store-inline
          - mountPath: "/opt/sonarqube/data/"
            name: sonar-data-new
          - mountPath: "/opt/sonarqube/extensions/plugins/"
            name: sonar-extensions-new2
          env:
          - name: "SONARQUBE_JDBC_USERNAME"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: username
          - name: "SONARQUBE_JDBC_PASSWORD"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: password
          - name: "SONARQUBE_JDBC_URL"
            valueFrom:
              configMapKeyRef:
                name: sonar-config
                key: url
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data-new
        persistentVolumeClaim:
          claimName: sonar-data-new
      - name: sonar-extensions-new2
        persistentVolumeClaim:
          claimName: sonar-extensions-new2
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
           secretProviderClass: "azure-kv-provider"

and this is my secret storage class:

kind: SecretProviderClass
metadata:
  name: azure-kv-provider
spec:
  provider: azure
  secretObjects:
   - data:
      - key: username
        objectName: username
      - key: password
        objectName: password
     secretName: test-secret
     type: Opaque
  parameters:
    usePodIdentity: "false"  
    useAssignedIdentity: "true"
    userAssignedIdentityID: "zzzz-zzzz-zzzz-zzzz-zzzz"
    keyvaultName: "dbkvtz" 
    cloudName: ""   
    objects:  |
      array:
        - |
          objectName: test
          objectType: secret
          objectAlias: username
          objectVersion: "" 
        - |
          objectName: test
          objectType: secret 
          objectAlias: password  
          objectVersion: ""       
    resourceGroup: "myresourcegroup"  
    subscriptionId: "yyyy-yyyy-yyyy-yyy-yyyy"       
    tenantId: "xxxx-xxxx-xxxx-xxx-xxxx" 

Where "zzzz-zzzz-zzzz-zzzz-zzzz" is the Client ID of the created Managed Identity.

In the Key Vault that i created "dbkvtz" i added through "Access Policy" the Managed Identity that i created. On the other hand in "Manage Identity" i am not able to add any role in "Azure Role Assignement" -- No role assignments found for the selected subscription. I don't know if it is necessary to add any role there.

The AKS cluster is setup for system assigned managed identity. I want to use Managed Identities to get access to the key vaults so i created a managed identity with client id "zzzz-zzzz-zzzz-zzzz-zzzz" (where is "z" a value from 0-9a-z).

I am not too familiar with keyvault integration in AKS so i am not sure if the config is ok.

I am getting this error:

kubectl describe pods:

  Normal   Scheduled    19m                  default-scheduler  Successfully assigned default/sonarqube-6bdb9cfc85-npbfw to aks-agentpool-16966606-vmss000000
  Warning  FailedMount  5m43s (x5 over 16m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline sonar-data-new sonar-extensions-new2 default-token-t45tw]: timed out waiting for the condition
  Warning  FailedMount  3m27s                kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[default-token-t45tw secrets-store-inline sonar-data-new sonar-extensions-new2]: timed out waiting for the condition
  Warning  FailedMount  71s (x2 over 10m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[sonar-data-new sonar-extensions-new2 default-token-t45tw secrets-store-inline]: timed out waiting for the condition
  Warning  FailedMount  37s (x17 over 19m)   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-6bdb9cfc85-npbfw, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set

logs az aks show -g RG -n SonarQubeCluster

{
  "aadProfile": null,
  "addonProfiles": {
    "azurepolicy": {
      "config": null,
      "enabled": true,
      "identity": {
        "clientId": "yy",
        "objectId": "zz",
        "resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
      }
    },
    "httpApplicationRouting": {
      "config": null,
      "enabled": false,
      "identity": null
    },
    "omsagent": {
      "config": {
        "logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
      },
      "enabled": true,
      "identity": {
        "clientId": "yy",
        "objectId": "zz",
        "resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
      }
    }
  },
  "agentPoolProfiles": [
    {
      "availabilityZones": [
        "1"
      ],
      "count": 2,
      "enableAutoScaling": false,
      "enableEncryptionAtHost": null,
      "enableFips": false,
      "enableNodePublicIp": null,
      "enableUltraSsd": null,
      "gpuInstanceProfile": null,
      "kubeletConfig": null,
      "kubeletDiskType": "OS",
      "linuxOsConfig": null,
      "maxCount": null,
      "maxPods": 110,
      "minCount": null,
      "mode": "System",
      "name": "agentpool",
      "nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
      "nodeLabels": {},
      "nodePublicIpPrefixId": null,
      "nodeTaints": null,
      "orchestratorVersion": "1.20.7",
      "osDiskSizeGb": 128,
      "osDiskType": "Managed",
      "osSku": "Ubuntu",
      "osType": "Linux",
      "podSubnetId": null,
      "powerState": {
        "code": "Running"
      },
      "provisioningState": "Succeeded",
      "proximityPlacementGroupId": null,
      "scaleDownMode": null,
      "scaleSetEvictionPolicy": null,
      "scaleSetPriority": null,
      "spotMaxPrice": null,
      "tags": null,
      "type": "VirtualMachineScaleSets",
      "upgradeSettings": null,
      "vmSize": "Standard_DS2_v2"
    }
  ],
  "apiServerAccessProfile": {
    "authorizedIpRanges": null,
    "enablePrivateCluster": false,
    "enablePrivateClusterPublicFqdn": null,
    "privateDnsZone": null
  },
  "autoScalerProfile": null,
  "autoUpgradeProfile": null,
  "azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
  "disableLocalAccounts": null,
  "diskEncryptionSetId": null,
  "dnsPrefix": "SonarQubeCluster-dns",
  "enablePodSecurityPolicy": null,
  "enableRbac": true,
  "extendedLocation": null,
  "fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
  "fqdnSubdomain": null,
  "httpProxyConfig": null,
  "id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
  "identity": {
    "principalId": "yy",
    "tenantId": "rr",
    "type": "SystemAssigned",
    "userAssignedIdentities": null
  },
  "identityProfile": {
    "kubeletidentity": {
      "clientId": "yy",
      "objectId": "zz",
      "resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
    }
  },
  "kubernetesVersion": "1.20.7",
  "linuxProfile": null,
  "location": "southcentralus",
  "maxAgentPools": 100,
  "name": "SonarQubeCluster",
  "networkProfile": {
    "dnsServiceIp": "10.0.0.10",
    "dockerBridgeCidr": "172.17.0.1/16",
    "loadBalancerProfile": {
      "allocatedOutboundPorts": null,
      "effectiveOutboundIPs": [
        {
          "id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
          "resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
        }
      ],
      "idleTimeoutInMinutes": null,
      "managedOutboundIPs": {
        "count": 1
      },
      "outboundIPs": null,
      "outboundIpPrefixes": null
    },
    "loadBalancerSku": "Standard",
    "natGatewayProfile": null,
    "networkMode": null,
    "networkPlugin": "kubenet",
    "networkPolicy": null,
    "outboundType": "loadBalancer",
    "podCidr": "10.244.0.0/16",
    "serviceCidr": "10.0.0.0/16"
  },
  "nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
  "podIdentityProfile": null,
  "powerState": {
    "code": "Running"
  },
  "privateFqdn": null,
  "privateLinkResources": null,
  "provisioningState": "Succeeded",
  "resourceGroup": "RG",
  "securityProfile": null,
  "servicePrincipalProfile": {
    "clientId": "msi"
  },
  "sku": {
    "name": "Basic",
    "tier": "Free"
  },
  "type": "Microsoft.ContainerService/ManagedClusters",
  "windowsProfile": null
}

Any idea of what is wrong?

Thank you in advance.