Complex AWS EKS / ENI / Route53 issue has us stumped. Need an expert.
Context:
We are working on dynamic game servers for a social platform (https://myxr.social) that transport game and video data using WebRTC / UDP SCTP/SRTP via https://MediaSoup.org
Each game server will have about 50 clients Each client requires 2-4 UDP ports
Our working devops strategy https://github.com/xr3ngine/xr3ngine/tree/dev/packages/ops
We are provisioning these game servers using Kubernetes and https://agones.dev
Mediasoup requires each server connection to a client be assigned individual ports. Each client will need two ports, one for sending data and one for receiving data; with a target maximum of about 50 users per server, this requires 100 ports per server be publicly accessible.
We need some way to route this UDP traffic to the corresponding gameserver. Ingresses appear to primarily handle HTTP(S) traffic, and configuring our NGINX ingress controller to handle UDP traffic assumes that we know our gameserver Services ahead of time, which we do not since the gameservers are spun up and down as they are needed.
Questions:
We see two possible ways to solve this problem.
Path 1
Assign each game server in the node group public IPs and then allocate ports for each client. Either IP v4 or v6. This would require SSL termination for IP ports in AWS. Can we use ENI and EKS to dynamically create and provision IP ports for each gameserver w/ SSL? Essentially expose these pods to the internet via a public subnet with them each having their own IP address or subdomain. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html We have been referencing this documentation trying to figure out if this is possible.
Path 2
Create a subdomain (eg gameserver01.gs.xrengine.io, etc) dynamically for each gameserver w/ dynamic port allocation for each client (eg client 1 30000-30004, etc). This seems to be limited by the ports accessible in the EKS fleet.
Are either of these approaches possible? Is one better? Can you give us some detail about how we should go about implementation?
The native way for receiving UDP traffic on Amazon EKS is by using a Kubernetes Service of type Loadbalancer with an extra annotation to get the NLB.
Example
apiVersion: v1
kind: Service
metadata:
name: my-game-app-service
annotation:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
selector:
app: my-game-app
ports:
- name: outgoing-port # choose your name
protocol: UDP
port: 9000 # choose your port
- name: incoming-port # choose your name
protocol: UDP
port: 9001 # choose your port
type: LoadBalancer