K
Q

Kubernetes nginx ingress path-based routing of HTTPS in AWS

10/31/2019

Question: Within Kubernetes, how do I configure the nginx ingress to treat traffic from an elastic load balancer as HTTPS, when it is defined as TCP?

I am working with a Kubernetes cluster in an AWS environment. I want to use an nginx ingress to do path-based routing of the HTTPS traffic; however, I do not want to do SSL termination or reencryption on the AWS elastic load balancer.

The desired setup is:

client -> elastic load balancer -> nginx ingress -> pod

Requirements:
1. The traffic be end-to-end encrypted.
2. An AWS ELB must be used (the traffic cannot go directly into Kubernetes from the outside world).

The problem that I have is that to do SSL passthrough on the ELB, I must configure the ELB as TCP traffic. However, when the ELB is defined as TCP, all traffic bypasses nginx.

As far as I can tell, I can set up a TCP passthrough via a ConfigMap, but that is merely another passthrough; it does not allow me to do path-based routing within nginx.

I am looking for a way to define the ELB as TCP (for passthrough) while still having the ingress treat the traffic as HTTPS.

I can define the ELB as HTTPS, but then there is a second, unnecessary negotiate/break/reencrypt step in the process that I want to avoid if at all possible.

-- Magua
amazon-elb
amazon-web-services
kubernetes
nginx

Similar Questions

helm test via Jenkins pipeline
Ephemeral Storage Garbage Collection
K8s Cluster over Azure using Kubeadm + Autoscaling using VMSS
k3s ctr images import to different dir

1 Answer

11/1/2019

To make it more clear I'll start from OSI model, which tells us that TCP is level 4 protocol and HTTP/HTTPS is level 7 protocol. So, frankly speaking HTTP/HTTP data is encapsulated to TCP data before doing rest levels encapsulations to transfer packet to another network device.

If you setup Classic (TCP) LoadBalancer it stops reading packet data after reading TCP part, which is enough to decide (according to LB configuration) to which IP address and to which IP port this data packet should be delivered. After that LB takes the TCP payload data and wrap it around with another TCP layer data and send it to the destination point (which in turn cause all other OSI layers applied).

To make your configuration works as expected, it's required to expose nginx-ingress-controller Pod using NodePort service. Then Classic ELB can be configured to deliver traffic to any cluster node to port selected for that NodePort service. Usually it is in between 30000 and 32767. Sou your LB pool will look like the following:

Let's imagine cluster nodes have IP addresses 10.132.10.1...10 and NodePort port is 30276.

ELB Endpoint 1:  10.132.10.1:30276
ELB Endpoint 2:  10.132.10.2:30276
...
ELB Endpoint 10: 10.132.10.10:30276

Note: In case of AWS ELB, I guess, nodes DNS names should be used instead of IP addresses.

So it should cause the following sequence of traffic distribution from a client to Kubernetes application Pod:

  1. Client sends TCP packet with HTTP/HTTPS request in payload to ELB_IP:ELB_port (a.b.c.d:80).
  2. ELB receives IP packet, analyze its TCP data, finds the appropriate endpoint from backend pool (whole list of Kubernetes cluster nodes), and creates another TCP packet with the same HTTP/HTTPS data inside, and also replaces destination IP and destination TCP port to cluster node IP and Service NodePort TCP port (l.m.n.k:30xxx) and then send it to the selected destination.
  3. Kubernetes node receives TCP packet and using the iptables rules changes the destination IP and destination port of the TCP packet again, and forward the packet (according to the Nodeport Service configuration) to destination pod. In this case it would be nginx-ingress-controller pod.
  4. Nginx-ingress-controller pod receives the TCP packet, and because according to TCP data it have to be delivered locally, extracts HTTP/HTTP data out of it and send the data (HTTP/HTTPS request) to Nginx process inside the Nginx container in the Pod,
  5. Nginx process in the container receives HTTP/HTTPS request, decrypt it (in case of HTTPS) and analyze all HTTP headers.
  6. According to nginx.conf settings, Nginx process change HTTP request and deliver it to the cluster service, specified for the configured host and URL path.
  7. Nginx process sends changed HTTP request to the backend application.
  8. Then TCP header is added to the HTTP request and send it to the backend service IP_address:TCP_port.
  9. iptables rules defined for the backend service, deliver packet to one of the service endpoints (application Pods).

Note: To terminate SSL on ingress controller you have to create SSL certificates that includes ELB IP and ELB FQDN in the SAN section.

Note: If you want to terminate SSL on the application Pod to have end to end SSL encryption, you may want to configure nginx to bypass SSL traffic.

Bottom line: ELB configured for delivering TCP traffic to Kubernetes cluster works perfectly with nginx-ingress controller if you configure it in the correct way.

In GKE (Google Kubernetes Engine) if you create a Service with type:LoadBalancer it creates you exactly TCP LB which forward traffic to a Service NodePort and then Kubernetes is responsible to deliver it to the Pod. EKS (Elastic Kubernetes Service) from AWS works in pretty much similar way.

-- VAS
Source: StackOverflow