K
Q

How to debug failing kubectl calls?

3/10/2017

I am on Suse 12.01 Enterprise and trying to get Minikube working. The VM works already and the minikube shell tool can communicate.

But the kubectl still can't talk to the kubernetes master. I'm trying to debug that, and the best way to get additional info seems to be running a random command with -v 9. Doing that I get the following output:

$ kubectl get pots -v 9
I0310 14:02:27.727767   29330 loader.go:354] Config loaded from file /home/D069407/.kube/config
I0310 14:02:27.728479   29330 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.5.3 (linux/amd64) kubernetes/029c3a4" https://192.168.99.104:8443/api
I0310 14:03:42.704009   29330 round_trippers.go:318] GET https://192.168.99.104:8443/api  in 74975 milliseconds
I0310 14:03:42.704037   29330 round_trippers.go:324] Response Headers:
I0310 14:03:42.704103   29330 helpers.go:221] Connection error: Get https://192.168.99.104:8443/api: Service Unavailable
F0310 14:03:42.704111   29330 helpers.go:116] Unable to connect to the server: Service Unavailable

Not much info, but my guess would be that curl -k -vvvv .... would give me more info. However, just doing the same curl as in the log results in an authentication error, since the api server does client auth (right?). So that doesn't work.

How can I continue to debug now? Is there a secret ninja way to give curl auth params without adding them to the shell call? Is kubectl actually doing another request and just prints the curl to the log to give some hints about what it's calling?

*edit: Discussing with a colleage we both agreed that this must be a minikube vm internal problem.

minikube logs are starting like this:

-- Logs begin at Fri 2017-03-10 12:43:34 UTC, end at Fri 2017-03-10 14:44:11 UTC. --
Mar 10 12:45:45 minikube systemd[1]: Starting Localkube...
Mar 10 12:45:45 minikube localkube[3496]: I0310 12:45:45.977140    3496 start.go:77] Feature gates:%!(EXTRA string=)
Mar 10 12:45:45 minikube localkube[3496]: localkube host ip address: 10.0.2.15
Mar 10 12:45:45 minikube localkube[3496]: I0310 12:45:45.981395    3496 server.go:215] Using iptables Proxier.
Mar 10 12:45:45 minikube localkube[3496]: W0310 12:45:45.981764    3496 server.go:468] Failed to retrieve node info: Get http://127.0.0.1:8080/api/v1/nodes/minikube: dial tcp 127.0.0.1:8080:
getsockopt: connection refused
Mar 10 12:45:45 minikube localkube[3496]: W0310 12:45:45.981879    3496 proxier.go:249] invalid nodeIP, initialize kube-proxy with 127.0.0.1 as nodeIP
Mar 10 12:45:45 minikube localkube[3496]: W0310 12:45:45.981947    3496 proxier.go:254] clusterCIDR not specified, unable to distinguish between internal and external traffic
Mar 10 12:45:45 minikube localkube[3496]: I0310 12:45:45.982082    3496 server.go:227] Tearing down userspace rules.
Mar 10 12:45:45 minikube localkube[3496]: Starting etcd...
Mar 10 12:45:45 minikube localkube[3496]: E0310 12:45:45.991070    3496 reflector.go:188] pkg/proxy/config/api.go:30: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?r
esourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Mar 10 12:45:45 minikube localkube[3496]: E0310 12:45:45.991108    3496 reflector.go:188] pkg/proxy/config/api.go:33: Failed to list *api.Endpoints: Get http://127.0.0.1:8080/api/v1/endpoint
s?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Mar 10 12:45:45 minikube localkube[3496]: name = kubeetcd
Mar 10 12:45:45 minikube localkube[3496]: data dir = /var/lib/localkube/etcd
Mar 10 12:45:45 minikube localkube[3496]: member dir = /var/lib/localkube/etcd/member
Mar 10 12:45:45 minikube localkube[3496]: heartbeat = 100ms
Mar 10 12:45:45 minikube localkube[3496]: election = 1000ms
Mar 10 12:45:45 minikube localkube[3496]: snapshot count = 10000
Mar 10 12:45:45 minikube localkube[3496]: advertise client URLs = http://0.0.0.0:2379
Mar 10 12:45:45 minikube localkube[3496]: initial advertise peer URLs = http://0.0.0.0:2380
Mar 10 12:45:45 minikube localkube[3496]: initial cluster = kubeetcd=http://0.0.0.0:2380
Mar 10 12:45:45 minikube localkube[3496]: starting member fcf2ad36debdd5bb in cluster 7f055ae3b0912328
Mar 10 12:45:45 minikube localkube[3496]: fcf2ad36debdd5bb became follower at term 0
Mar 10 12:45:45 minikube localkube[3496]: newRaft fcf2ad36debdd5bb [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Mar 10 12:45:45 minikube localkube[3496]: fcf2ad36debdd5bb became follower at term 1
Mar 10 12:45:46 minikube localkube[3496]: starting server... [version: 3.0.14, cluster version: to_be_decided]
Mar 10 12:45:46 minikube localkube[3496]: Starting apiserver...
Mar 10 12:45:46 minikube localkube[3496]: Starting controller-manager...
Mar 10 12:45:46 minikube localkube[3496]: Starting scheduler...
Mar 10 12:45:46 minikube localkube[3496]: Starting kubelet...
Mar 10 12:45:46 minikube localkube[3496]: added member fcf2ad36debdd5bb [http://0.0.0.0:2380] to cluster 7f055ae3b0912328
Mar 10 12:45:46 minikube localkube[3496]: Starting proxy...
Mar 10 12:45:46 minikube localkube[3496]: Starting storage-provisioner...

inside minikube ssh the service api works, though. Checked with curl 127.0.0.1:8080/api and receiving nice json.

*edit: based on feedback some more infos.

curl inside the minikube vm:

$ curl localhost:8080
{
"paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/apps",
    "/apis/apps/v1beta1",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v2alpha1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1alpha1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1alpha1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/extensions/third-party-resources",
    "/healthz/poststarthook/rbac/bootstrap-roles",
    "/logs",
    "/metrics",
    "/swaggerapi/",
    "/ui/",
    "/version"
]
}$ curl localhost:8080/api
{
"kind": "APIVersions",
"versions": [
    "v1"
],
"serverAddressByClientCIDRs": [
    {
    "clientCIDR": "0.0.0.0/0",
    "serverAddress": "10.0.2.15:8443"
    }
]
}

kube config (outside vm):

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/<user>/.minikube/ca.crt
    server: https://192.168.99.104:8443
name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
    client-certificate: /home/<user>/.minikube/apiserver.crt
    client-key: /home/<user>/.minikube/apiserver.key
-- erikbwork
curl
kubectl
kubernetes
minikube
sles

Similar Questions

Failed to start Kubernetes API Server
Often restarting of the pods because of networking issues
Kubernetes not restarting failed pods on another docker node
Automatic cluster setup and app deployment on GCE Kubernetes

1 Answer

3/10/2017

You said you tried curl 127.0.0.1:8080/api, but regarding to the logs it tries to connect via https. So you should try curl https://127.0.0.1:8080/api.

I searched the source code for the term Service Unavailable and it for example the error description for a HTTP 503 return code.

Out of my guts, I assume you will get an HTTP 503, when you try to curl https://127.0.0.1:8080/api.

Edit: Since you use minikube, we can assume that this is running correctly. In this case it is more likely that there is a configuration issue. Your logs say that kubectl tries to connect to localhost. AFAIK this is the default host. When you start minikube it doesn't run on the host, rather than in a virtual machine. Therefore localhost looks wrong and you should take a look at the kube config.

-- svenwltr
Source: StackOverflow