K
Q

Newer versions of Minikube don't allow Pods to use their own Services

9/7/2017

I have a Pod that occasionally needs to call itself by its own host-name. I have a Deployment set up that ultimately creates the Pod, and a Service in place. I'm able to use the service name from a different Pod within my cluster, but the Pod cannot call itself using its host-name. This is something that works with Minikube v0.17.1 with Kubernetes version 1.5.3, but upgrading either Minikube or the Kubernetes version seems to break things. This also works on our deployed / server version of Kubernetes.

Did something change that I need to take account of in my Pod / Service setup? How do I go about getting past this?

How things should work

Running Minikube version 0.17.1:

Start Minikube:

$ minikube start
Starting local Kubernetes cluster...
Starting VM...
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

$ minikube version
minikube version: v0.17.1

Deploy minimal test images (yaml definition below):

kubectl apply -f deploy_python.yaml

Exec into the python2 image, and verify connection to python image:

$ kubectl exec -it python2-1281934109-k015g bash
root@python2-1281934109-k015g:/# curl python:12345 --connect-timeout 10
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="boot/">boot/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="lib64/">lib64/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>

Exec into python image and verify connection to self:

$ kubectl exec -it python-2555691705-5j0f9 bash
root@python-2555691705-5j0f9:/# curl python:12345 --connect-timeout 10
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="boot/">boot/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="lib64/">lib64/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>

That's a success. By creating a Deployment and a Service, I'm able to make requests to the referenced Pod from any other Pod in the cluster.

The way it works in newer versions

(Stop and delete running Minikube.)
Start Minikube, specifying Kubernetes version 1.7.0:

$ minikube start --kubernetes-version=v1.7.0
Starting local Kubernetes cluster...
Starting VM...
SSH-ing files into VM...
Downloading localkube binary
137.48 MB / 137.48 MB [============================================] 100.00% 0s
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-30T10:17:58Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Deploy minimal test images (yaml definition below):

$ kubectl apply -f deploy_python.yaml
service "python" created
deployment "python" created
deployment "python2" created

Exec into python 2 image, and verify connection to python image:

$ kubectl exec -it python2-380393367-ztgkq bash
root@python2-380393367-ztgkq:/# curl python:12345 --connect-timeout 10
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="boot/">boot/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="lib64/">lib64/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>

Exec into python image and attempt connection to self:

$ kubectl exec -it python-2168884431-gls2j bash
root@python-2168884431-gls2j:/# curl python:12345 --connect-timeout 10
curl: (28) Connection timed out after 10000 milliseconds

Yaml file, deploy_python.yaml:

---
apiVersion: v1
kind: Service
metadata:
  name: python
spec:
  selector:
    app: python
  ports:
  - port: 12345
    targetPort: 12345
    name: http
---    
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: python
  labels:
    app: python
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: python
    spec:
      containers:
      - image: python
        name: python
        command: ["python"]
        args: ["-m", "http.server", "12345" ]
        ports:
        - containerPort: 12345
---    
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: python2
  labels:
    app: python2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: python2
    spec:
      containers:
      - image: python
        name: python2
        command: ["python"]
        args: ["-m", "http.server", "12345" ]
        ports:
        - containerPort: 12345
-- GamerJosh
kubernetes
minikube

Similar Questions

Minikube can't pull image from local registry
Sharing a persistent volume between pods in Kubernetes
Redis db permission issue while running GitLab
How to keep a Kubernetes pod running when main process is not blocking?

1 Answer

9/15/2017

This is a known bug in newer versions of kubernetes / minikube and is being tracked here:

https://github.com/kubernetes/kubernetes/issues/20475

The current accepted workaround (https://github.com/kubernetes/kubernetes/issues/20475#issuecomment-190995739) is to ssh into your minikube vm and run a specific ip link command.

minikube ssh
sudo ip link set docker0 promisc on

I have tested this workaround and it seems to work for me.

-- GamerJosh
Source: StackOverflow