Getting GKE secrets back even after deleting the KMS keys used for encryption

6/20/2019

I followed this document to create a GKE cluster (1.13.6-gke.6) with --database-encryption-key flag giving a KMS key for enabling Application-layer Secrets Encryption.

I created a secret using the following command:

kubectl create secret generic dev-db-secret --from-literal=username=someuser --from-literal=password=somepass

So if my assumption is correct, these secrets are stored encrypted using the KMS key provided by me while creating the cluster. However, even after I have destroyed all the versions of the used key, I am able to see the secret stored inside the GKE etcd using kubectl get secret dev-db-secret -o yaml as well as I am able to see them in a pod created using the below manifest:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: dev-db-secret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: test-secret
            key: password
  restartPolicy: Never

If I exec into the above pod and do echo SECRET_USERNAME and echo SECRET_PASSWORD I get the username and password printed on my console in plain text.

Is this the way the encryption supposed to work? If yes, where is the encryption happening exactly? What am I doing wrong? Are the secrets really encrypted?

-- Amit Yadav
google-cloud-platform
google-kubernetes-engine
kubernetes-secrets
kubernetes-security

1 Answer

6/20/2019

I'm not 100% sure, but I think those keys are cached so it's probably will take a while before the decryption will fail. This is the case for Azure, I guess it's similar for GKE.

BTW you might want to read how to protect the manifest files so you can store them on Git. I wrote a blog post describing some of the options you can use.

-- Omer Levi Hevroni
Source: StackOverflow