Just getting started with Kubernetes. I cannot seem to connect pods running on different nodes to communicate with each other.

I set up a Kubernetes Cluster with Calico networking on three AWS EC2 instances (one master, two workers all with src/dest check disabled as described by the Calico website). Each instance is using the same Security Group with all TCP/UDP/ICMP ports open for 10.0.0.0/8 and 192.168.0.0/16 to make sure there is no blocked ports inside my cluster.

using a vanilla repo install

~$ sudo apt-get install -y docker.io kubelet kubeadm kubectl
~$ sudo kubeadm init --pod-network-cidr=192.168.0.0/16  

and basic Calico install

~$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

joined two worker nodes to the cluster

 sudo kubeadm join <Master IP>:6443 --token <Token>  --discovery-token-ca-cert-hash sha256:<cert hash> 

Once up and running, I created three replicas for testing:

~$ kubectl run pingtest --image=busybox --replicas=3 -- sleep infinity

two on the first node and one on the second node

~$ kubectl get pod -l run=pingtest -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP               NODE              NOMINATED NODE   READINESS GATES
pingtest-7689dd958f-9mfgl   1/1     Running   0          15m   192.168.218.65   ip-10-78-31-198   <none>           <none>
pingtest-7689dd958f-l288v   1/1     Running   0          15m   192.168.218.66   ip-10-78-31-198   <none>           <none>
pingtest-7689dd958f-z2l97   1/1     Running   0          15m   192.168.237.65   ip-10-78-11-83    <none>           <none>

log into a shell on the first pod

~$ kubectl exec -ti pingtest-7689dd958f-9mfgl /bin/sh

When I ping pods on the same node everything works

/ # ping 192.168.218.66 -c 2
PING 192.168.218.66 (192.168.218.66): 56 data bytes
64 bytes from 192.168.218.66: seq=0 ttl=63 time=0.105 ms
64 bytes from 192.168.218.66: seq=1 ttl=63 time=0.078 ms

but when I ping a pod on another node, no response

/ # ping 192.168.237.65 -c 2
PING 192.168.237.65 (192.168.237.65): 56 data bytes

--- 192.168.237.65 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

What am I missing? What is preventing communication between the pods on different nodes?