A little background on this issue: i set up a raspberry pi k8s cluster (1 master 3 workers) as part of learning more about kubernetes & networking.

I then decided to set up pivpn on one of the nodes too so that i can connect to cluster without setting up port forwarding on router for every single node, but ran into a few issues which i spent more than a week troubleshooting but cant find any answers.

I have set up pivpn before but it was on a raspberry pi that isnt running kubernetes and have never encountered any of the issues mentioned below.

Here are the 2 issues that i will really appreciate to get advice on:

1. Laptop unable to connect to pivpn if on similar network. however it is able to connect if laptop connects via tethered connection from mobile phone. i do not remember facing this issue previously when setting up pivpn though.

Error shown in logs:

2020-04-25 14:26:56.479178 TLS: Initial packet from [AF_INET]xxx.xx.xxx.xxx:1194, sid=4e337288 178e6c09
2020-04-25 14:27:57.159491 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2020-04-25 14:27:57.159621 TLS Error: TLS handshake failed
2020-04-25 14:27:57.160296 SIGUSR1[soft,tls-error] received, process restarting

2. After connecting to pivpn via mobile tethering, laptop do not have internet access and also not able to ping any other devices from within the network (i.e router, 3 other raspberry pi nodes)

I checked iptables and route on node running pivpn. Seems like issue could be with Docker related iptable rules, in particular the FORWARD chain, which based on my understanding is not forwarding any packets hence no internet access. im not too sure about this too.

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             10.8.0.0/24          ctstate RELATED,ESTABLISHED /* openvpn-forward-rule */
ACCEPT     all  --  10.8.0.0/24          anywhere             /* openvpn-forward-rule */
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
ACCEPT     all  --  10.244.0.0/16        anywhere
ACCEPT     all  --  anywhere             10.244.0.0/16
ACCEPT     all  --  10.17.0.0/16         anywhere
ACCEPT     all  --  anywhere             10.17.0.0/16

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

I have also provided the configs for pivpn setup.

server.conf

dev tun
proto udp
port 1194
ca /etc/openvpn/easy-rsa/pki/ca.crt
cert /etc/openvpn/easy-rsa/pki/issued/kube2_148c8722-ed05-4c6a-94ef-15a1322a78f4.crt
key /etc/openvpn/easy-rsa/pki/private/kube2_148c8722-ed05-4c6a-94ef-15a1322a78f4.key
dh /etc/openvpn/easy-rsa/pki/dh2048.pem
topology subnet
server 10.8.0.0 255.255.255.0
# Set your primary domain name server address for clients
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"
# Prevent DNS leaks on Windows
push "block-outside-dns"
# Override the Client default gateway by using 0.0.0.0/1 and
# 128.0.0.0/1 rather than 0.0.0.0/0. This has the benefit of
# overriding but not wiping out the original default gateway.
push "redirect-gateway def1"
client-to-client
client-config-dir /etc/openvpn/ccd
keepalive 15 120
remote-cert-tls client
tls-version-min 1.2
tls-auth /etc/openvpn/easy-rsa/pki/ta.key 0
cipher AES-256-CBC
auth SHA256
user openvpn
group openvpn
persist-key
persist-tun
crl-verify /etc/openvpn/crl.pem
status /var/log/openvpn-status.log 20
status-version 3
syslog
verb 3
#DuplicateCNs allow access control on a less-granular, per user basis.
#Remove # if you will manage access by user instead of device.
#duplicate-cn
# Generated for use by PiVPN.io

client.ovpn

client
dev tun
proto udp
remote xxx.xx.xx.xxx 1194
resolv-retry infinite
nobind
key-direction 1
remote-cert-tls server
tls-version-min 1.2
verify-x509-name kube2_148c8722-ed05-4c6a-94ef-15a1322a78f4 name
cipher AES-256-CBC
auth SHA256
auth-nocache
verb 3
<ca>
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
xxx
-----END PRIVATE KEY-----
</key>
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
xxx
-----END OpenVPN Static key V1-----
</tls-auth>