I followed this document to create a GKE cluster (1.13.6-gke.6) with --database-encryption-key
flag giving a KMS key for enabling Application-layer Secrets Encryption.
I created a secret using the following command:
kubectl create secret generic dev-db-secret --from-literal=username=someuser --from-literal=password=somepass
So if my assumption is correct, these secrets are stored encrypted using the KMS key provided by me while creating the cluster. However, even after I have destroyed all the versions of the used key, I am able to see the secret stored inside the GKE etcd
using kubectl get secret dev-db-secret -o yaml
as well as I am able to see them in a pod created using the below manifest:
apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: mycontainer
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: dev-db-secret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: test-secret
key: password
restartPolicy: Never
If I exec into the above pod and do echo SECRET_USERNAME
and echo SECRET_PASSWORD
I get the username and password printed on my console in plain text.
Is this the way the encryption supposed to work? If yes, where is the encryption happening exactly? What am I doing wrong? Are the secrets really encrypted?
I'm not 100% sure, but I think those keys are cached so it's probably will take a while before the decryption will fail. This is the case for Azure, I guess it's similar for GKE.
BTW you might want to read how to protect the manifest files so you can store them on Git. I wrote a blog post describing some of the options you can use.