I've created 2 kubernetes
cluster within the same VPC
in the same region in AWS.
The first cluster is dedicated to my micro services.Let's name it "MS" The second one is dedicated to vault and its highly available storage (consul). Let's name it "V"
The question is how i can get access to the secrets I've created in "V" cluster, from the containers in "MS" cluster ?
What I've tried so far:
I started by creating a new service account in "MS" cluster which authenticates with the review token API.
Then I had to extract the token reviewer JWT, Kuberenetes CA certificate and the Kubernetes host from "MI" cluster
Then I switched to for "V" cluster context to enabled and create a new kubernetes auth method attached to that service account.
From there I don't know what to do and I'm not sure that method really works when using 2 different clusters?
Service account:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: consul
labels:
app: consul
rules:
- apiGroups: [""]
resources:
- pods
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: consul
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: consul
subjects:
- kind: ServiceAccount
name: consul
namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: consul
labels:
app: consul
Export token review variables from "MI" cluster
export VAULT_SA_NAME=$(kubectl get sa postgres-vault -o jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
export K8S_HOST=$(kubectl exec consul-consul-0 -- sh -c 'echo $KUBERNETES_SERVICE_HOST')
Create kubernetes auth method
vault auth enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="https://$K8S_HOST:443" \
kubernetes_ca_cert="$SA_CA_CRT"
I expect to gain access to the secrets stored in vault from my micros services. Though I'm not sure that method works when vault is deployed in a dedicated cluster.
I guess there might be something else to join both clusters? May be using consul?
You are 80% there. the next steps are: 1. Run a deployment with the correct service account 2. Login/Authenticate to vault using Kubernetes authentication method and get a relevant vault token. 3. Retrieve secrets.
This is an example of adding the service account to your deployment:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: xxx
labels:
app: xxx
spec:
replicas: 1
template:
metadata:
...
spec:
serviceAccountName: vault-auth
...
To login using kube auth see here. I would suggest that you take a look at this project for a concrete example.
Once you get token with the correct permissions you can use vault cli/ rest api to access secrets.