K
Q

Proxy kubernetes outgoing traffic via Proxy (CNTLM)

8/15/2018

I am trying to proxy all outgoing kubernetes traffic via a CNTLM which runs on the host.

A little bit of context: Currently i am trying to setup a kubernetes cluster on VM to use as fast deployment solution for projects. Sadly all outbound traffic which is not in the corporate network has to be authenticated via NTLM. On the host where the kubernetes cluster is running the cntlm server is running on port 3128.

So what i want to achieve is that all traffic from the pods redirected to host:3128. What i have thought of are the following ideas:

  • Modifying the iptables of the host to reroute traffic. This worked quite well for docker based containers without kubernetes. (See https://hub.docker.com/r/ncarlier/redsocks/). With the container you could define your proxy and additionally a whitelist which should not be applied for proxying. Is this also possible for kubernetes?

  • The other idea would be to start a pod to which all traffic from the other pods will be routed. And this pod acts as CNTLM proxy. Not sure if this possible.

Machine setup:

Kubernetes is working as expected in the network without external access.

Thanks for any help :)

UPDATE:

What i already tried like Artem Golenyaev mentioned:

  • Editing in docker proxy stuff for using the proxy. (reload + restart done)
  • Editing .bashrc + sourcing for applying the proxy-

Content of .bashrc

export http_proxy=http://d050alapi138:3128
export HTTP_PROXY=$http_proxy
export https_proxy=$http_proxy
export HTTPS_PROXY=$http_proxy
printf -v lan '%s,' 53.190.251.237
printf -v service '%s,' 10.96.0.{1..253}
printf -v pool '%s,' 192.168.0.{1..253}
export no_proxy="${lan%,},${service%,},${pool%,},127.0.0.1";
export NO_PROXY=$no_proxy

Content of /etc/systemd/system/docker.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://d050alapi138:3128" "NO_PROXY=localhost,d050alapi138"

Content of /etc/systemd/system/docker.service.d/https-proxy.conf:

[Service]
Environment="HTTPS_PROXY=http://d050alapi138:3128" "NO_PROXY=localhost,d050alapi138"

Testing proxy is working in bash:

d050alapi138:~ # curl google.de
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.de/">here</A>.
</BODY></HTML>

Now script for creating cluster:

kubeadm init --apiserver-advertise-address=53.190.251.237 --service-cidr=10.96.0.0/16 --pod-network-cidr=192.168.0.0/24

export KUBECONFIG=/etc/kubernetes/admin.conf

kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

kubectl taint nodes --all node-role.kubernetes.io/master-

The node starts up and i can deploy ymls and so and when i try to test the internet connection inside a pod:

d050alapi138:~ # kubectl run my-shell2 --rm -i --tty --image ubuntu -- bash
If you don't see a command prompt, try pressing enter.
root@my-shell2-66df6fcdf4-4jhc8:/# apt-get update
0% [Connecting to archive.ubuntu.com (2001:67c:1360:8001::21)] [Connecting to security.ubuntu.com (2001:67c:1560:8001::11)]^C

It doesn't work. First when i would manually set the proxy envs inside the container it works

root@my-shell2-66df6fcdf4-4jhc8:/# export http_proxy=http://d050alapi138:3128
root@my-shell2-66df6fcdf4-4jhc8:/# export https_proxy=$http_proxy
root@my-shell2-66df6fcdf4-4jhc8:/# apt-get update
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/universe Sources [11.5 MB]
0% [3 InRelease gpgv 74.6 kB] [4 Sources 0 B/11.5 MB 0%] [Waiting for headers]^C

Maybe this helps to understand my problem.

-- Eduard Marbach
corporate
docker
kubernetes
proxy

Similar Questions

GKE does not scale to/from 0 when autoscaling enabled
Limiting the maximum number of pods per deployment within a namespace / cluster
In AKS how do I limit access to the cluster DNS to internal VMs
Migrating Cockroach DB from local machine to GCP Kubernetes Engine
What are the Kubernetes alternatives logging to ELK/EFK stack for Node.js apps

1 Answer

8/16/2018

You can try to use the common solution called "Kubernetes behind the corporate proxy."

First, you need to add proxy settings to Docker on all Nodes to allow it downloading images. Create or modify /etc/systemd/system/docker.service.d/http-proxy.conf file with the following lines (of course, you need to change addresses, ports, and networks in the following example):

  • For HTTP proxy:

    [Service]    
Environment="HTTP_PROXY=<http://proxy.example.com>:<proxy_port>/" "NO_PROXY=localhost,127.0.0.1,<docker-registry.somecorporation.com>"

  • For HTTPS proxy

    [Service]    
Environment="HTTPS_PROXY=<https://proxy.example.com>:<proxy_port>/" "NO_PROXY=localhost,127.0.0.1,<docker-registry.somecorporation.com>"

Then, you need to restart the Docker daemon:

systemctl daemon-reload
systemctl restart docker

Second, you need to add a proxy setting on all your Nodes into .bashrc to forward required traffic from these Nodes to the proxy. Here is the example:

export http_proxy=<http://proxy.example.com>:<proxy_port>/
export HTTP_PROXY=$http_proxy
export https_proxy=<https://proxy.example.com>:<proxy_port>/
export HTTPS_PROXY=$http_proxy
printf -v lan '%s,' localip_of_machine 
printf -v pool '%s,' 192.168.0.{1..253}
printf -v service '%s,' 10.96.0.{1..253}
export no_proxy="${lan%,},${service%,},${pool%,},127.0.0.1";
export NO_PROXY=$no_proxy

Also, you need to use your own settings for http_proxy, https_proxy, and no_proxy.

For more information, you can visit the following links:

-- Artem Golenyaev
Source: StackOverflow