I am using the ingress-nginx system https://github.com/kubernetes/ingress-nginx. I'm making extensive use of this project. Jenkins, Consul, Prometheus and more are working just fine using the exact same ingress config as appended to the end.
I am able to access my vault pod directly through port-forwarding with kubectl. But when I attempt to access it over my nginx-ingress, I am returned a 503
kubectl port-forward vault-vault-f9778f86d-srr9n 8200:8200 -n vault
curl 127.0.0.1:8200/v1/1
{"errors":["Vault is sealed"]}
➜ vault curl -L vault.me.com/v1/1
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.13.8</center>
</body>
</html>
Looking at the logs I see the following in response to the vault.me.com curl
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:42:56 +0000] "GET / HTTP/1.1" 308 187 "-" "curl/7.43.0" 77 0.000 [] - - - -
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:42:56 +0000] "GET / HTTP/1.1" 503 213 "-" "curl/7.43.0" 77 0.000 [] - - - -
Where as if I try to access my consul backend, I see the following.
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:34 +0000] "GET / HTTP/1.1" 308 187 "-" "curl/7.43.0" 78 0.000 [consul-consul-consul-8500] - - - -
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:39 +0000] "GET / HTTP/1.1" 308 187 "-" "curl/7.43.0" 78 0.000 [consul-consul-consul-8500] - - - -
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:39 +0000] "GET / HTTP/1.1" 301 39 "-" "curl/7.43.0" 78 0.002 [consul-consul-consul-8500] 10.233.114.4:8500 39 0.002 301
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:39 +0000] "GET /ui/ HTTP/1.1" 200 30178 "-" "curl/7.43.0" 81 0.001 [consul-consul-consul-8500] 10.233.82.19:8500 30178 0.001 200
I'm not entire sure whats going on, nor am I clear on how to debug this. I've spent the two hours reading through the source on this tool, but I'm not seeing much of anything.
I read somewhere that any response that isn't 2xx or 3xx fails nginx's backend health check and will be removed from the pool of backends to be routed to. That would manifest as the 503 that I am seeing. Its true that vault's / does return a non 2xx or 3xx so that would make sense.
vault curl -v 127.0.0.1:8200
* Rebuilt URL to: 127.0.0.1:8200/
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8200 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8200
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Cache-Control: no-store
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 19 Apr 2018 20:46:40 GMT
< Content-Length: 19
<
404 page not found
Is there a way to change the URL that nginx tries to healthcheck, or disable the healthchecking entirely
https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/ From nginx's docs it says if there is a single server in a backend group, it should never be getting marked as unavailable, regardless of return code.
Whats going on?
Note that if there is only a single server in a group, the fail_timeout and max_fails parameters are ignored and the server is never marked unavailable.
k get svc -n vault
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vault-vault ClusterIP 10.233.47.151 <none> 8200/TCP 1h
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx-internal
name: vault
namespace: vault
spec:
rules:
- host: vault.me.com
http:
paths:
- backend:
serviceName: vault-vault
servicePort: 8200
path: /
tls:
- hosts:
- me.com
- vault.me.com
secretName: wildcard-secret
Unsealing the vault through the port-forward method will change the state of the kubernetes pod level healthchecking, which is what is queried by ingress-nginx to determine if a backend is valid or not.
Once the vault is unsealed you can access the backend as you expect.
At the time of writing, the vault CLI does not appear to respect http 308 redirects. Which gives you a very unhelpful error about golang's parser library. I wasted several hours on this. I hope this information helps someone.