Assume that when you create nodes in rancher you don't fill private IP fields, so nodes will connect through public ip, does it have any security issue?
I am not sure which version of Rancher you are using. In 1.6, the cross-host communication happens over IPSec tunnels, whereas in 2.x, depending on the network plugin, it could be mostly VXLAN.
I wouldn't use VXLAN overlay over the internet. IPSec is designed for secure communications.
Also, I would use some form of protection using features like Security Groups to shut down the use of other ports.
If you really don't have a use case to use public IP addresses, you should override the defaults to use private IP addresses.