kubernetes : error listing nodes

7/21/2018

I provisioned kubernetes cluster using kops:

kops create cluster \
    --zones ap-southeast-1a \
    --dns-zone=${DNS_ZONE} \
    --node-count 3 \
    --vpc=${VPC_ID} \
    --subnets=${SUBNET_ID} \
    ${NAME}

Ec2 instances are provisioned and I can see them in console.

DNS_ZONE is abc.in, a public hosted zone in route53

api.sp-southeast-1.es.k8s.abc.in. A 54.123.123.123
api.internal.sp-southeast-1.es.k8s.abc.in. A 172.123.123.123
etcd-a.internal.sp-southeast-1.es.k8s.abc.in. A 172.123.123.123
etcd-events-a.internal.sp-southeast-1.es.k8s.abc.in. A 172.123.123.123 

I've gone through the stack overflow discussions on the same problem.

I'm using 3 t2.micro instances for master and 3 t2.medium instances for Nodes

Inbound rules for master node are :

Custom TCP Rule TCP 4003 - 65535 sg-0559ecf7ab (nodes.sp-southeast-1.es.k8s.abc.in) 
Custom TCP Rule TCP 2382 - 4000 sg-0559674a0d3ab (nodes.sp-southeast-1.es.k8s.abc.in) 
All traffic All All sg-0ecc38031f44 (masters.sp-southeast-1.es.k8s.abc.in) 
SSH TCP 22 0.0.0.0/0 
Custom UDP Rule UDP 1 - 65535 sg-0557674a0d3ab (nodes.sp-southeast-1.es.k8s.abc.in) 
Custom TCP Rule TCP 1 - 2379 sg-0557674a0d3ab (nodes.sp-southeast-1.es.k8s.abc.in) 
Custom TCP Rule TCP 443 0.0.0.0/0

when I enter

kops validate cluster
> unexpected error during validation: error listing nodes: 
Get https://api.sp-southeast-1.es.k8s.abc.in/api/v1/nodes: 
dial tcp 54.123.123.123:443: i/o timeout
-- prranay
devops
kops
kubectl
kubernetes

1 Answer

8/17/2018

I have upgraded my production kube cluster and I encounter the same problem. We use AWS cloud.

The problem was due to the incorrect dns configuration (AWS Route 53) of the cluster api, maybe is a bug into kops?

To solve I manually set the correct ip address for the kube api and then the api is now reachable.

UPDATE: I found that the problem was due to an incorrect kops upgrade, the kubelet responsible for the network service was not run correctly. The cluster use a 1.10 kubernetes but aws ec2 images are at 1.9. Force cluster to 1.9.8 version, the cluster upgrade run correctly and the kubelet network service updates dns correctly.

-- Pistis Valentino
Source: StackOverflow