I am running a HashiCorp Vault stateless set in K8s with 3 pods on three nodes.
After deployment I manually unseal Vault. Vault then stays unsealed all the time.
The problem is when one of the nodes is restarted, the Vault pod restarts in unsealed mode. Is there any way to automatically unseal the Vault node by server-server-communication with one of the already unsealed pods?
I do not want to have to unseal the Vault pods manually when my Kubernetes environment gets updated and all nodes get restarted (rolling update - one by one).
I also do not want to store the unseal keys in a K8s secret or even a file as this makes encryption of my secrets useless.
This is my yaml:
apiVersion: v1
kind: Service
metadata:
name: vault
spec:
clusterIP: None
ports:
- name: http
port: 8200
- name: server
port: 8201
selector:
xyz.service: vault
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: vault
labels:
xyz.service: vault
spec:
serviceName: "vault"
selector:
matchLabels:
xyz.service: vault
replicas: 3
template:
metadata:
labels:
xyz.service: vault
spec:
imagePullSecrets:
- name: reg-dhc-xyzoms-pull-secret
securityContext:
runAsUser: 100
fsGroup: 1000
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: xyz.service
operator: In
values:
- vault
topologyKey: kubernetes.io/hostname
containers:
- name: vault
image: vault:0.11.0
resources:
requests:
memory: "100Mi"
env:
- name: SKIP_SETCAP
value: dontcare
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: "status.podIP"
- name: "VAULT_CLUSTER_ADDR"
value: "https://$(POD_IP):8201"
ports:
- name: http
containerPort: 8200
protocol: "TCP"
- name: server
containerPort: 8201
protocol: "TCP"After some more digging I learned, what I want is impossible. Whenever a Vault instance is restarted it will be unsealed first and there is no way to unseal it automatically using Vault-own techniques.
You can find a lot of "vault-unsealer" implementations in GitHub and Docker store which try to fill this gap by regularly checking the Vault pods state and unsealing it if necessary.
It is suggested to use an K8s readinessprobe to avoid that services access a sealed Vault pod.
As there is no official "vault-unsealer" image, the public implementations must be used with caution. I ended up writing my own "vault-unsealer" to avoid security flaws and licensing problems.
My solution is a sidecar-container with each Vault pod. The unseal keys first have to be entered once manually with kubectl exec ... at one sidecar. The sidecars regularly check all Vault pods and communicate the unseal keys to the other sidecar if sealed. If a sidecar receives unseal keys, they are stored in memory and are used to unseal its own Vault instance.
kubect aply -f vault.yaml -> vault-0 startingkubectl exec vault-0 -c sidecar ... to enter unseal keys -> vault-0 sidecar unseals vault-0 and is ready