I have a single node kubernetes deployment running on a home server, on which I have several services running. Since it's a small local network, I wanted to block off a portion of the local address range that the rest of my devices use for pod ips, and then route to them directly.

For example, if I have a web server running, instead of exposing port 80 as an external port and port forwarding from my router to the worker node, I would be able to port forward directly to the pod ip.

I haven't had much luck finding information on how to do this though, is it possible?

I'm new to kubernetes so I am sure I am leaving out important information, please let me know so I can update the question.

I got this working by using the macvlan CNI plugin from the reference plugins. Using kubeadm to set up the cluster, these plugins are already installed and the cluster will be configured to use them. The only thing to do is drop in a cni.conf (in /etc/cni/net.d). Mine looks like this

{
    "name": "net",
    "type": "macvlan",
    "mode": "bridge",
    "master": "eno1",
    "ipam": {
        "type": "host-local",
        "ranges": [[{ 
            "subnet": "10.0.0.0/8",
            "gateway": "10.0.0.1",
            "rangeStart": "10.0.10.2",
            "rangeEnd": "10.0.10.254" 
        }]],
        "routes": [
            { "dst": "0.0.0.0/0" }
        ]
    }
}

Putting this in place is all that is needed for coredns to start up and any pods you run will have ips from the range defined in the config. Since this is on the same subnet as the rest of my lan, I can freely ping these containers and my router even lets me play with their settings since they have mac addresses (if you dont want this use ipvlan instead of macvlan, you'll still be able to ping and port forward and everything, your router just wont be enumerating all the devices since they dont have hardware addresses).

Couple of caveats:

1. Services won't work since they're all "fake" (e.g. they dont have interfaces its all iptables magic that makes them work). There's probably a way to make them work but it wasn't worth it for my use case

2. For whatever reason the DNS server keeps revering to 10.96.0.1. I have no idea where it got that address from, but I have been working around it by defining dnsPolicy: None and setting dnsConfig.nameservers[0] to my routers IP. There's probably a better solution for for it.

3. You should run kubeadm with --service-cidr 10.0.10.0/24 --pod-network-cidr 10.0.10.0/24 or it seems like kubelet (or something) doesn't know how to talk to the pods. I actually don't know if --service-cidr matters but it seems like a good idea

4. Out of the box, your pods wont be able to talk to the master since they are using macvlan devices enslaving its ethernet and for whatever reason macvlan doesn't let you talk between host and guest devices. As you can imagine this isnt a good thing. Solution is to manually add a macvlan device on the host with the same subnet as your pods.

5. It seems like even ports you don't expose from the pod are usable from the lan devices (which isnt cool), probably since the iptables rules think that anything on the lan is cluster-internal. I haven't put much time into debugging this.

This is probably some kind of cardinal sin for people used to using kubernetes in production, but its kind of cool and useful for a home setup, though it certainly feels like a hack sometimes.