I've been toying around with kubernetes and have run into an issue. The core of my problem is that while I can access the services on the master node by curling localhost, attempting to access the same via the public ip and port doing the same on another machine (or web browser) hangs forever.
I've configured the cluster using a terraform script (runs as root):
#!/bin/bash -v
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>startup_log.out 2>&1
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
cat << EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y docker-ce
apt-mark hold docker-ce
apt-get install -y kubelet kubeadm kubectl kubernetes-cni
apt-mark hold kubelet kubeadm kubectl
kubeadm init --token=${k8stoken} --pod-network-cidr=10.244.0.0/16
mkdir -p /home/ubuntu/.kube
cp -i /etc/kubernetes/admin.conf /home/ubuntu/.kube/config
chown -R $(id -u ubuntu):$(id -g ubuntu) /home/ubuntu/.kube/
usermod -aG docker ubuntu
echo "net.bridge.bridge-nf-call-iptables=1" | tee -a /etc/sysctl.conf
sysctl -p
runuser -l ubuntu -c '\
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
'
After which I SSH to the master node (works fine) and run:
$ git clone https://github.com/linuxacademy/robot-shop.git
$ kubectl create namespace robot-shop
$ kubectl -n robot-shop create -f ~/robot-shop/K8s/descriptors/
$ ubuntu@ip-10-0-100-167:~$ kubectl -n robot-shop get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cart ClusterIP 10.105.152.135 <none> 8080/TCP 15s
catalogue ClusterIP 10.97.111.197 <none> 8080/TCP 15s
dispatch ClusterIP None <none> 55555/TCP 15s
mongodb ClusterIP 10.107.178.183 <none> 27017/TCP 15s
mysql ClusterIP 10.110.254.52 <none> 3306/TCP 15s
payment ClusterIP 10.99.195.138 <none> 8080/TCP 15s
rabbitmq ClusterIP 10.99.70.232 <none> 5672/TCP,15672/TCP 15s
ratings ClusterIP 10.98.80.21 <none> 80/TCP 15s
redis ClusterIP 10.101.232.84 <none> 6379/TCP 15s
shipping ClusterIP 10.106.246.97 <none> 8080/TCP 15s
user ClusterIP 10.109.120.146 <none> 8080/TCP 15s
web NodePort 10.97.162.113 <none> 8080:30080/TCP 15s
Services appear to startup fine but I can't access anything on this host externally. I've tried following this debugging guide, and identified a discrepancy that when I type:
$ iptables-save | grep
I get nothing. But the doc doesn't make a recommendation as to a fix. Has anyone seen this issue before? I'm concerned my config file has a problem with it but I can't figure out what!
As @Ashworth mentioned in the comment issue has been solved after appropriate service port enabled in the particular security group for the relevant master node instance.