Grant permission to use the key in GKE

1/13/2020

I'm trying to provision an encrypted disk for GKE dynamically. But I really don't understand below part.

Grant permission to use the key
You must assign the Compute Engine service account used by nodes in your cluster the Cloud KMS CryptoKey Encrypter/Decrypter role. This is required for GKE Persistent Disks to access and use your encryption key.

The Compute Engine service account's name has the following format:

service-[PROJECT_NUMBER]@compute-system.iam.gserviceaccount.com

Does it really necessary to grant "Cloud KMS CryptoKey Encrypter/Decryter" to Compute Engine Service account? Can I create a new SA and grant this role to it? The description said, the SA used by nodes. So I'm wondering if I can create a new SA and grant Cloud KMS role then use this SA to spin up GKE cluster. Then I think it should be available to provision encrypted disks for GKE.

official document below:

dynamically_provision_an_encrypted

-- Dust break
encryption
google-cloud-platform
google-kubernetes-engine
kubernetes

1 Answer

1/13/2020

I tried to follow this documentation step by step:

  1. create gke cluster (check Kubernetes Compatibility compatibility, I decided to stick with 1.14 this time), key-ring and key

  2. deploy CSI driver to the cluster

    2.1. download driver $git clone https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver /PATH/gcp-compute-persistent-disk-csi-driver

    2.2. configure variables for your project in /PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh

    2.3. create service account with /PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh

2.4. configure variables for driver deployment in /PATH/gcp-compute-persistent-disk-csi-driver/deploy/kubernetes/deploy-driver.sh and /PATH/gcp-compute-persistent-disk-csi-driver/deploy/kubernetes/deploy-driver.shinstall-kustomize.sh

2.5. deploy CSI driver (I stick with stable version)

$./deploy-driver.sh
  1. enable the Cloud KMS API

  2. assign the Cloud KMS CryptoKey Encrypter/Decrypter role (roles/cloudkms.cryptoKeyEncrypterDecrypter) to the Compute Engine Service Agent (service-[PROJECT_NUMBER]@compute-system.iam.gserviceaccount.com)

  3. create StorageClass

    $cat storage.yaml    
    apiVersion: storage.k8s.io/v1beta1
    kind: StorageClass
    metadata:
      name: csi-gce-pd
    provisioner: pd.csi.storage.gke.io
    parameters:
      type: pd-standard
      disk-encryption-kms-key: projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY
    
    
    $kubectl describe storageclass csi-gce-pd
    Name:            csi-gce-pd
    IsDefaultClass:  No
    Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{},"name":"csi-gce-pd"},"parameters":{"disk-encryption-kms-key":"projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY","type":"pd-standard"},"provisioner":"pd.csi.storage.gke.io"}
    
    Provisioner:           pd.csi.storage.gke.io
    Parameters:            disk-encryption-kms-key=projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY,type=pd-standard
    AllowVolumeExpansion:  <unset>
    MountOptions:          <none>
    ReclaimPolicy:         Delete
    VolumeBindingMode:     Immediate
    Events:                <none>
  4. create persistent volume

    $kubectl apply -f pvc.yaml
    persistentvolumeclaim/podpvc created

    $kubectl describe pvc podpvc Name: podpvc Namespace: default StorageClass: csi-gce-pd Status: Bound Volume: pvc-b383584a-32c5-11ea-ad6e-42010a9c007d Labels: Annotations:

       kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"podpvc","namespace":"default"},"spec":{"accessModes...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      6Gi
    Access Modes:  RWO
    VolumeMode:    Filesystem
    Mounted By:    <none>
    Events:
      Type    Reason                 Age                From                                                                                                           Message
      ----    ------                 ----               ----                                                                                                           -------
      Normal  Provisioning           31m                pd.csi.storage.gke.io_gke-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  External provisioner is provisioning volume for claim "default/podpvc"
      Normal  ExternalProvisioning   31m (x2 over 31m)  persistentvolume-controller                                                                                    waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
      Normal  ProvisioningSucceeded  31m                pd.csi.storage.gke.io_gke-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  Successfully provisioned volume pvc-b383584a-32c5-11ea-ad6e-42010a9c007d

And it's successfully provisioned.

Then I removed Cloud KMS CryptoKey Encrypter/Decrypter role from the Compute Engine Service Agent and persistent volume created at step 6 and tried again:

$kubectl apply -f pvc.yaml                                
persistentvolumeclaim/podpvc created

$kubectl describe pvc podpvc
Name:          podpvc
Namespace:     default
StorageClass:  csi-gce-pd
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
             {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"podpvc","namespace":"default"},"spec":{"accessModes...
           volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
Type     Reason                Age                   From                                                                                                           Message
----     ------                ----                  ----                                                                                                           -------
Normal   Provisioning          2m15s (x10 over 11m)  pd.csi.storage.gke.io_gke-serhii-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  External provisioner is provisioning volume for claim "default/podpvc"
Warning  ProvisioningFailed    2m11s (x10 over 11m)  pd.csi.storage.gke.io_gke-serhii-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  failed to provision volume with StorageClass "csi-gce-pd": rpc error: code = Internal desc = CreateVolume failed to create single zonal disk "pvc-b1a238b5-35fa-11ea-bec8-42010a9c01e6": failed to insert zonal disk: unkown Insert disk error: googleapi: Error 400: Cloud KMS error when using key projects/serhii-test-prj/locations/europe-west3/keyRings/SERHII-TEST-KEY-RING/cryptoKeys/SERHII-TEST-KEY: Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/serhii-test-prj/locations/europe-west3/keyRings/SERHII-TEST-KEY-RING/cryptoKeys/SERHII-TEST-KEY' (or it may not exist)., kmsPermissionDenied
Normal   ExternalProvisioning  78s (x43 over 11m)    persistentvolume-controller                                                                                    waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator

and persistent volume stayed in pending status.

And, as you can see, in the documentation it's necessary:

Grant permission to use the key

You must assign the Compute Engine service account used by nodes in your cluster the Cloud KMS CryptoKey Encrypter/Decrypter role. This is required for GKE Persistent Disks to access and use your encryption key.

and it's not enough to create service account with /PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh provided by CSI driver.

EDIT Please notice that:

For CMEK-protected node boot disks, this Compute Engine service account is the account which requires permissions to do encryption using your Cloud KMS key. This is true even if you are using a custom service account on your nodes.

So, there's no way to use only service account in this case without Compute Engine service account, because CMEK-protected persistent volumes are managed by GCE, not by GKE. Meanwhile, you can provide only necessary perdition to your custom service account to improve security of your project.

-- Serhii Rohoza
Source: StackOverflow