I got an issue when I try to access an exposed kubernetes service through browser. Below is my Environment.
created two ubuntu EC2 instances(with all ports open in security group) and installed all kubernetes related tools like kubectl, kubeadm, docker, calico network.
created nginx pod, scaled it to 3 and exposed it with type LoadBalancer. When I curl from master or worker node to the exposed nginx it works fine(with public or private ip). But it does not work if i curl from outside. The request is timed out. I tried to delete service and expose it again with NodePort. But still I could not access from outside. I ensured the security group allows ingress. Is there a way to debug why it cannot be accessed from outside or I am missing something
I am not running cloud controller manager but kube-controller-manager. Will this be an issue.?
below is the output of all kubernetes components
ubuntu@ip-172-31-29-98:~$ kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/nginx-6f858d4d45-2wtlh 1/1 Running 0 51m
default pod/nginx-6f858d4d45-5dkws 1/1 Running 0 51m
default pod/nginx-6f858d4d45-h9cwg 1/1 Running 0 51m
kube-system pod/calico-etcd-82xkv 1/1 Running 1 18h
kube-system pod/calico-kube-controllers-74b888b647-prr2q 1/1 Running 1 18h
kube-system pod/calico-node-kbckk 2/2 Running 4 17h
kube-system pod/calico-node-n5zhr 2/2 Running 3 18h
kube-system pod/coredns-78fcdf6894-qjhlq 1/1 Running 1 18h
kube-system pod/coredns-78fcdf6894-sm7c9 1/1 Running 1 18h
kube-system pod/etcd-ip-172-31-29-98 1/1 Running 1 18h
kube-system pod/kube-apiserver-ip-172-31-29-98 1/1 Running 1 18h
kube-system pod/kube-controller-manager-ip-172-31-29-98 1/1 Running 1 18h
kube-system pod/kube-proxy-jxg88 1/1 Running 1 18h
kube-system pod/kube-proxy-knx59 1/1 Running 1 17h
kube-system pod/kube-scheduler-ip-172-31-29-98 1/1 Running 1 18h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 18h
default service/nginx LoadBalancer 10.99.144.149 <pending> 80:31808/TCP 45m
kube-system service/calico-etcd ClusterIP 10.96.232.136 <none> 6666/TCP 18h
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 18h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-etcd 1 1 1 1 1 node-role.kubernetes.io/master= 18h
kube-system daemonset.apps/calico-node 2 2 2 2 2 <none> 18h
kube-system daemonset.apps/kube-proxy 2 2 2 2 2 beta.kubernetes.io/arch=amd64 18h
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
default deployment.apps/nginx 3 3 3 3 51m
kube-system deployment.apps/calico-kube-controllers 1 1 1 1 18h
kube-system deployment.apps/calico-policy-controller 0 0 0 0 18h
kube-system deployment.apps/coredns 2 2 2 2 18h
NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/nginx-6f858d4d45 3 3 3 51m
kube-system replicaset.apps/calico-kube-controllers-74b888b647 1 1 1 18h
kube-system replicaset.apps/calico-policy-controller-55b469c8fd 0 0 0 18h
kube-system replicaset.apps/coredns-78fcdf6894 2 2 2 18hEdit 1: Tried to do the same in GCloud. It is same in GCloud too. The nginx Service is accessible through private/public ip inside the nodes. But when I curl from outside, it does not work. I spin up a simply python server in one of the node and I am able to access the python server from outside. But only the services exposed through kubernetes are not curl-able from outside.
I think I am missing some fundamental understanding about kubernetes networking(especially in cloud). Can I get any help from experts?
How did you create your cluster? kubeadm? You need some custom configs for your cluster to run with AWS. For example your kube-controller-manager and kube-apiserver need to have the option --cloud-provider=aws. Same for all your kubelets.
I'd recommend using kops if you don't want to deal with this.
The internet firewall on my network was blocking the access to specific ports. There was no issues with the Cloud Provider Firewall or Instance Firewall.