I have created a dedicated kubernetes cluster for consul and vault in AWS.

My cluster has 3 vault instances and all of them are currently running which means, I've unsealed vault in every single instance

Vault node instances are exposed through an AWS NLB.

When logging via web ui or command line I get:

That issue has to do with the following post https://github.com/hashicorp/vault/issues/1337

* Note :* I've found out that if I seal 2 of vault instance and leave one unsealed, I'm able to log in.

So is vault supposed to run only one instance at the same time ?

Will it unseal automatically one of the other isntance if the current leader fails ?

Please check my vault-service.yaml

kind: Service
apiVersion: v1
metadata:
  name: vault-lb
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  selector:
    app: vault
  ports:
  - name: vault-port
    port: 443
    targetPort: 8200
    protocol: TCP

Pleaze find a piece of vault-statefulset.yaml

- name: vault
        image: vault:latest
        env:
        - name: LOAD_BALANCER_ADDR
          valueFrom:
            configMapKeyRef:
              name: vault
              key: load_balancer_address
        - name: POD_IP_ADDR
          valueFrom:
            fieldRef:
              fieldPath: "status.podIP"
        - name: VAULT_ADDR
          valueFrom:
            configMapKeyRef:
              name: vault
              key: vault-addr
        - name: VAULT_CLUSTER_ADDR
          value: "https://$(POD_IP):8201"
        - name: VAULT_CONSUL_KEY
          valueFrom:
            secretKeyRef:
              name: vault-consul-key
              key: consul-key
        - name: VAULT_LOCAL_CONFIG
          value:
            api_addr     = "https://$(LOAD_BALANCER_ADDR)"
            cluster_addr = "https://$(POD_IP_ADDR):8201"
            log_level = "warn"
            ui = true
            backend "consul" {
              address = "127.0.0.1:8500"
              redirect_addr = "https://$(LOAD_BALANCER_ADDR):8200"
              cluster_addr = "https://$(VAULT_1_SERVICE_HOST):$(VAULT_1_SERVICE_PORT_BACKENDPORT)"
              token = "$(VAULT_CONSUL_KEY)"
              disable_registration = "false"
            }

            listener "tcp" {
              address     = "127.0.0.1:8200"
              tls_disable = "true"
            }
            listener "tcp" {
              address       = "$(POD_IP_ADDR):8200"
              tls_cert_file = "/etc/vault/tls/tls.crt"
              tls_key_file  = "/etc/vault/tls/tls.key"
              tls_disable_client_certs = true
            }
        args:
        - "server"
        ports:
        - name: vault-port
          containerPort: 8200
          protocol: TCP
        - name: cluster-port
          containerPort: 8201
          protocol: TCP
        readinessProbe:
          httpGet:
            path: "/v1/sys/health?standbyok=true"
            port: 8200
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
        securityContext:
          capabilities:
            add:
            - IPC_LOCK
        volumeMounts:
        - name: vault-tls
          mountPath: /etc/vault/tls
        - name: log-storage
          mountPath: /vault/logs
      - name: consul-agent-client
        image: "consul:1.5.0"
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
          limits:
            cpu: 100m
            memory: 200Mi
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: GOSSIP_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: consul
              key: gossip-encryption-key
        args:
        - "agent"
        - "-data-dir=/tmp/consul"
        - "-encrypt=$(GOSSIP_ENCRYPTION_KEY)"
        - "-domain=cluster.local"
        - "-retry-join=consul-0.consul.$(NAMESPACE).svc.cluster.local"
        - "-config-dir=/etc/consul"
        - "-node=$(HOSTNAME)"