How to access vault secrets from another kubernetes cluster when vault is deployed on a dedicated cluster?

5/14/2019

I've created 2 kubernetes cluster within the same VPC in the same region in AWS.

The first cluster is dedicated to my micro services.Let's name it "MS" The second one is dedicated to vault and its highly available storage (consul). Let's name it "V"

The question is how i can get access to the secrets I've created in "V" cluster, from the containers in "MS" cluster ?

What I've tried so far:

  1. I started by creating a new service account in "MS" cluster which authenticates with the review token API.

  2. Then I had to extract the token reviewer JWT, Kuberenetes CA certificate and the Kubernetes host from "MI" cluster

  3. Then I switched to for "V" cluster context to enabled and create a new kubernetes auth method attached to that service account.

From there I don't know what to do and I'm not sure that method really works when using 2 different clusters?

Service account:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: consul
  labels:
    app: consul
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul
subjects:
  - kind: ServiceAccount
    name: consul
    namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul
  labels:
    app: consul

Export token review variables from "MI" cluster

export VAULT_SA_NAME=$(kubectl get sa postgres-vault -o jsonpath="{.secrets[*]['name']}")

export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)

export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)

export K8S_HOST=$(kubectl exec consul-consul-0 -- sh -c 'echo $KUBERNETES_SERVICE_HOST')

Create kubernetes auth method

vault auth enable kubernetes
vault write auth/kubernetes/config \
  token_reviewer_jwt="$SA_JWT_TOKEN" \
  kubernetes_host="https://$K8S_HOST:443" \
  kubernetes_ca_cert="$SA_CA_CRT"

I expect to gain access to the secrets stored in vault from my micros services. Though I'm not sure that method works when vault is deployed in a dedicated cluster.

I guess there might be something else to join both clusters? May be using consul?

-- jaybe78
consul
devops
hashicorp-vault
kubernetes

1 Answer

5/14/2019

You are 80% there. the next steps are: 1. Run a deployment with the correct service account 2. Login/Authenticate to vault using Kubernetes authentication method and get a relevant vault token. 3. Retrieve secrets.

This is an example of adding the service account to your deployment:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: xxx
  labels:
    app: xxx
spec:
  replicas: 1
  template:
    metadata:
      ...
    spec:
      serviceAccountName: vault-auth
  ...

To login using kube auth see here. I would suggest that you take a look at this project for a concrete example.

Once you get token with the correct permissions you can use vault cli/ rest api to access secrets.

-- Amityo
Source: StackOverflow