I wrote a script that starts minikube and uses kubectl port-forward
to forward a local port to one of the services running in the kubernetes cluster. It runs kubectl port-forward
in the background so it can run a web-server on localhost with access to a service in the cluster.
There's a race-condition where kubectl port-forward
fails because the kubelet isn't ready to accept connections yet.
error upgrading connection: error dialing backend: dial tcp 127.0.1.1:10250: getsockopt: connection refused
Short of putting a sleep
in my script, how might I check whether the kubelet is ready before running the port-forward?
Since the error message indicates we expect the kubelet to be listening for tcp connections on port 10250, I thought I could do something like this:
ss --listening --tcp --numeric --processes \
'( sport = 10250 )' \
| grep --quiet kubelet
But I found that even once the kubelet is ready, that ss
command doesn't find it. I don't know if this is due to the 127.0.1.1
ip address or if there's something else I'm missing.
The main script, run-server
, looks like:
#! /usr/bin/env bash
LOCAL_PORT=5678
cleanup() {
local pid=$(ss --listening --tcp --processes \
"( sport = ${LOCAL_PORT} )" \
| grep -oP 'pid=\K\d+')
[ -n "$pid" ] && kill "$pid"
}
trap 'cleanup' EXIT
minikube start
run-port-forward $LOCAL_PORT &
FOO_SERVICE_URL=localhost:${LOCAL_PORT} bin/rails server
and run-port-forward
looks like
#! /usr/bin/env bash
LOCAL_PORT=$1
pod=$(kubectl --context=minikube get pod \
| grep -m 1 -oP "foo-service\S*")
port=$(kubectl --context=minikube get pod ${pod} -o json \
| jq -r .spec.containers[0].ports[0].containerPort)
kubectl --context=minikube port-forward \
${pod} ${LOCAL_PORT}:${port}
But I found that even once the kubelet is ready, that ss command doesn't find it. I don't know if this is due to the 127.0.1.1 ip address or if there's something else I'm missing.
minikube runs on a VM on your machine which has its own interface and IP address. You can retrieve it using minikube ip
So if you did:
curl -X GET http://$(minikube ip):10250/healthz
Assuming you've configured the following options:
--healthz-bind-address 0.0.0.0
The IP address for the healthz server to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 127.0.0.1)
--healthz-port int32
The port of the localhost healthz endpoint (set to 0 to disable) (default 10248)
Though I haven't yet found a way to check for kubelet readiness using kubectl, I came up with a workaround that performs a few retries.
This is the first time I've used recursion in a bash script. I got a kick out of that.
try_port_forward() {
retries=$1
if (( retries > 0 )); then
sleep 1
kubectl --context=minikube port-forward \
${pod} ${LOCAL_PORT}:${port} \
2>/dev/null ||
try_port_forward $(( retries - 1 ))
fi
}
try_port_forward 10