I have a Django application running in a container that I would like to probe for readiness. The kubernetes version is 1.10.12. The settings.py specifies to only allow traffic from a specific domain:
ALLOWED_HOSTS = ['.example.net']
If I set up my probe without setting any headers, like so:
containers:
- name: django
readinessProbe:
httpGet:
path: /readiness-path
port: 8003
then I receive a 400 response, as expected- the probe is blocked from accessing readiness-path
:
Invalid HTTP_HOST header: '10.5.0.67:8003'. You may need to add '10.5.0.67' to ALLOWED_HOSTS.
I have tested that I can can successfully curl the readiness path as long as I manually set the host headers on the request, so I tried setting the Host headers on the httpGet, as partially documented here, like so:
readinessProbe:
httpGet:
path: /readiness-path
port: 8003
httpHeaders:
- name: Host
value: local.example.net:8003
The probe continues to fail with a 400.
Messing around, I tried setting the httpHeader with a lowercase h, like so:
readinessProbe:
httpGet:
path: /django-admin
port: 8003
httpHeaders:
- name: host
value: local.example.net:8003
Now, the probe actually hits the server, but it's apparent from the logs that instead of overwriting the HTTP_HOST header with the correct value, it has been appended, and fails because the combined HTTP_HOST header is invalid:
Invalid HTTP_HOST header: '10.5.0.67:8003,local.example.net:8003'. The domain name provided is not valid according to RFC 1034/1035
Why would it recognize the header here and add it, instead of replacing it?
One suspicion I am trying to validate is that perhaps correct handling of host headers was only added to the Kubernetes httpHeaders spec after 1.10. I have been unable to find a clear answer on when host headers were added to Kubernetes- there are no specific headers described in the API documentation for 1.10.
Is it possible to set host headers on a readiness probe in Kubernetes 1.10, and if so how is it done? If not, any other pointers for getting this readiness probe to correctly hit the readiness path of my application?
Update:
I have now tried setting the value without a port, as suggested by a commenter:
httpHeaders:
- name: Host
value: local.acmi.net.au
The result is identical to setting the value with a port. With a capital H the host header value is not picked up at all, with a lowercase h the host header value is appended to the existing host header value.
This problem is fixed in Kubernetes 1.2.3+ — see kubernetes/kubernetes#24288. I've configured one of my deployments in a similar way (for similar reasons). This is for a Craft CMS instance which doesn't require authentication at the /admin/login
url:
readinessProbe:
httpGet:
path: /admin/login
port: 80
httpHeaders:
- name: Host
value: www.example.com
timeoutSeconds: 5
Steven Shaw's answer was very useful, but the winning combination wound up finally being:
readinessProbe:
httpGet:
path: /readiness-path/ # << NO REDIRECTS ON THIS PATH
port: 8003
httpHeaders:
- name: host
value: local.example.net # << NO PORT ON THE HOST VALUE
After some log contemplation I finally realized that the path I was probing for readiness was redirecting from /readiness-path
to /readiness-path/
. Once I supplied the trailing slash to sidestep the redirect, it started working. From this I conclude that the redirects weren't preserving the httpHeaders being set on the probe request.