I am trying to build a networking application in a Kubernetes cluster which processes packets between two servers and offers a security service on the traffic going across the application. Following points summarize how this application works:

1. Server1 and Server2 connect to the network gateway application over IPsec tunnels.

2. The packets sent by Server1 are encapsulated in the IPsec tunnel and are destined to Server2, that is, the destination IP address in the inner packet is Server2 IP address.

3. IPsec server runs in a Pod in the cluster. This has been defined as a Kubernetes service IPSEC.

4. I have created another Kubernetes service, called TELEMETRY, which works on the network packets. This service collects some telemetry data from the packets and optionally applies some policies on the packets.

5. The decrypted packets received on this IPSEC pod having destination IP address same as Server2 IP address, need to be forwarded to TELEMETRY service pod.

6. The TELEMETRY service pod processes the packet and is supposed to forward it back to IPSEC pod so that the packet can be forwarded to the actual destination Server2 over an IPsec tunnel.

7. Once this will work, I also plan to add couple of services after TELEMETRY, so that the TELEMETRY service pod will forward the packet to those services. The last service in chain will forward it back to the IPSEC pod.

I am facing challenges in forwarding the IP packets from one pod to another. I want to forward packets that are not destined to any service or pod in the cluster from one pod to another pod. I explored Flannel, but it does not forward the packets on overlay if the destination pod is running on the same node. It doesn't appear to be good solution from performance perspective due to multiple packet copies between user and kernel space.

It would be really a great help if someone can suggest a good solution for forwarding the IP packets from one pod to another pod in my application.