I am trying to run Vault as a StatefulSet on Kubernetes.
I have a working consul cluster based on this: https://github.com/kelseyhightower/consul-on-kubernetes
My sts file for Vault looks like this:
kind: StatefulSet
metadata:
name: vault
spec:
serviceName: vault
replicas: 2
template:
metadata:
labels:
app: vault
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- vault
topologyKey: kubernetes.io/hostname
containers:
- name: vault
image: "vault:0.9.0"
ports:
- containerPort: 8200
name: http
- containerPort: 8201
name: backend
args:
- "server -config=/vault/config/vault-server.json"
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- name: config
mountPath: /vault/config
- name: tls
mountPath: /etc/tls
volumes:
- name: config
configMap:
name: vault
- name: tls
secret:
secretName: vault
My config file looks like this
{
"disable_mlock": true,
"listener": [
{
"tcp": {
"tls_disable": true
}
}
],
"storage": {
"consul": {
"address": "consul.default.svc.cluster.local:8500",
"path": "vault",
"token": "7e21f292-e7e7-f879-210c-4af2ae483cac"
}
}
}
When I apply the StatefulSet, I get a bind error
Error initializing listener of type tcp: listen tcp 127.0.0.1:8200: bind: address already in use
I have tried adding a listener with 127.0.0.1 and 0.0.0.0 with different ports. The pod is reading the config file because I was getting TLS warnings until I disabled.
Any ideas on what is bound to localhost on the pod? Any troubleshooting help would be appreciated
You can try this
Replace this:args: - "server -config=/vault/config/vault-server.json"
Add this in your yaml filecommand: ["vault", "server", "-config", "/vault/config/config.json"]
The issue was the Docker container starts vault in dev mode
From https://github.com/hashicorp/docker-vault/blob/master/0.X/Dockerfile#L69
# By default you'll get a single-node development server that stores everything
# in RAM and bootstraps itself. Don't use this configuration for production.
CMD ["server", "-dev"]
I added/changed the cmd and argument lines in the statefulSet yaml to
command: ["vault", "server"]
args:
- "-config=/vault/config/vault-server.json"
This gets rid of dev mode and uses server mode.
Please note this is not a production ready example, it is just for learning