I'm quite lost and I need some clear guidance.
I have created 2 kubernetes clusters on aws in a VPC. Each cluster has their own subnets.
The first cluster "MI" is dedicated to micro services and applications needing vault secrets
The second one "VA" is dedicated to consul/vault.
Obviously the idea is to make the "MI" cluster talks to vault in the "VA" cluster. In the init container of each micro service I would get a vault token like that:
$(curl -sS --request POST $VAULT_ADDRESS/v1/auth/kubernetes/login -H "Content-Type: application/json" -d '{"role":"$VAULT_POLICY","jwt":"$JWT"}' --insecure | jq -r 'if .errors then . else .auth.client_token end')
To achieve that, even though it's not recommended, I need a load balancer as my cluster nodes are not reachable, they only have private ip... right ?
On this matter, the documentation says:
Sometimes clients use load balancers as an initial method to access one of the Vault servers, but actually have direct access to each Vault node. In this case, the Vault servers should actually be set up as described in the above section, since for redirection purposes the clients have direct access.
In my case bearing in mind, vault nodes are behind VPC, I guess that paragraph does not apply to me as I would not be able to reach those address directly.
Then it says:
However, if the only access to the Vault servers is via the load balancer, the api_addr on each node should be the same: the address of the load balancer. Clients that reach a standby node will be redirected back to the load balancer; at that point hopefully the load balancer's configuration will have been updated to know the address of the current leader. This can cause a redirect loop and as such is not a recommended setup when it can be avoided.
I guess I'm in that situation since my vault instances will only be available through the ELB right? To make this work I understand I have to set
api_addr = "https://$(LOAD_BALANCER_ADDR):8200"
is there any way to fix the redirect loop ?
I've seen that configuration using ELB also set up global ip for it. I don't understand why static ip matters ? Isn't DNS sufficient ?
On AWS you cannot create global ip for LB, but since kubernetes 1.9 you can use network LB which has static IP?
So would you advise classic LB or latest NLB ?