WebRTC on Kubernetes cluster
I have a dedicated server build of a in-development realtime multiplayer game that I'd like to run on a Kubernetes cluster. The game uses WebRTC RTCDataChannels for network communication as it targets browsers. When running the game server on a Kubernetes node players can connect directly to it if I configure my Pods with hostNetwork: true. In the absence of hostNetwork: true players can still connect directly but only if they're behind well behaving NATs, if they're not then the only option for a successful connection is to introduce a TURN server to relay the traffic.
The reason for this appears to be that the game server runs on a Kubernetes node behind what is essentially a symmetric NAT i.e. the NATs mapping behaviour is address and port dependent. I've confirmed this by firing two STUN messages at different STUN servers from the same UDP socket within the container - the binding responses have the same public IP but different ports.
This NAT behaviour impacts WebRTC by reducing the success rate of players connecting directly to the game server in the absence of using a TURN server to relay traffic. Now I'd very much like to avoid relaying traffic through a TURN server where possible due to the added latency - it's not overly desirable for a realtime multiplayer game.
I'd have liked to just set hostNetwork: true and be done with it but I'm considering using Agones which doesn't support it (as it takes away the ability to securely do sidecars).
So I'm wondering if I have any other options (ideally without introducing yet another server to relay traffic through) to tweak this NAT behaviour as I don't believe it's feasible to try to forward the full range of random UDP ports that WebRTC is going to try to use for communication?
Update: Reading through the TURN RFC and I'm now thinking the below diagrammed setup may be possible using a TURN server running in the same container as the dedicated game server (thus latency introduction should be minimal).
The game clients will act as the TURN Clients i.e. create the allocation on the TURN server, etc.
The Game Server will act as the peer with which the clients want to communicate. The Game Server itself wouldn't need to be a TURN client and if my understanding is correct not even need to know that the TURN server is a TURN server.
Similar Questions
2 Answers
It is possible to serve many WebRTC connections using a single listening UDP/TCP port. You demux using the ICE ufrag and pwd and then route using the remote 3-tuple.
I am not sure what WebRTC implementation you are using, but with Pion you can enable this with SetICEUDPMux the actual code is in pion/ice. If we have seen this address before we demux it, otherwise we try to look up the ICE values.
I also have seen people go down the TURN route, I would recommend against it though. When you add one more piece that can drop traffic under heavy load it can be frustrating debugging.
Indeed, a TURN server can be deployed in the backend to allow exposing only a single UDP and/or TCP port to WebRTC clients. It's not the classic usage but it's something I've seen in production a couple times. Only the client requires the TURN server to be set up, the game server will communicate directly with the TURN server as it'll relay on the local network address. Additionally, you should force ICE relay on the client.
Another option could be to set a port range in the WebRTC agent configuration and forward the port range on the NAT. In that case, instead of setting a STUN server, you can manually override emitted host candidates with the external address to optimize connection establishment.