I'm using private Docker registry addon in my kubernetes cluster, and I would like to expose port 5000 on each node to pull image from localhost:5000 easily. So I placed a pod manifest file /etc/kubernetes/manifests/kube-registry-proxy.manifest on every node to start a local proxy for port 5000. It works when I manually deployed kubernetes on bare metal ubuntu few months ago, but failed when I try kargo, the port 5000 not listening.

I'm using kargo with calico network plugin, the docker registry's configurations are:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-system-kube-registry-pv
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /registry

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: kube-registry-pvc
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2.5.1
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        persistentVolumeClaim:
          claimName: kube-registry-pvc

apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    protocol: TCP

I have created a pod manifest file /etc/kubernetes/manifests/kube-registry-proxy.manifest before run kargo:

apiVersion: v1
kind: Pod
metadata:
  name: kube-registry-proxy
  namespace: kube-system
spec:
  containers:
  - name: kube-registry-proxy
    image: gcr.io/google_containers/kube-registry-proxy:0.3
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
    env:
    - name: REGISTRY_HOST
      value: kube-registry.kube-system.svc.cluster.local
    - name: REGISTRY_PORT
      value: "5000"
    - name: FORWARD_PORT
      value: "5000"
    ports:
    - name: registry
      containerPort: 5000
      hostPort: 5000

kube-registry-proxy is running on all nodes, but nothing listen on port 5000. Some output:

ubuntu@k8s15m1:~$ kubectl get all --all-namespaces | grep registry-proxy
kube-system   po/kube-registry-proxy-k8s15m1             1/1       Running             1          1h
kube-system   po/kube-registry-proxy-k8s15m2             1/1       Running             0          1h
kube-system   po/kube-registry-proxy-k8s15s1             1/1       Running             0          1h

ubuntu@k8s15m1:~$ docker ps | grep registry
756fcf674288        gcr.io/google_containers/kube-registry-proxy:0.3      "/usr/bin/run_proxy"     19 minutes ago      Up 19 minutes                           k8s_kube-registry-proxy.bebf6da1_kube-registry-proxy-k8s15m1_kube-system_a818b22dc7210ecd31414e328ae28e43_7221833c

ubuntu@k8s15m1:~$ docker logs 756fcf674288 | tail
waiting for kube-registry.kube-system.svc.cluster.local to come online
starting proxy

ubuntu@k8s15m1:~$ netstat -ltnp | grep 5000

ubuntu@k8s15m1:~$ curl -v localhost:5000/v1/
*   Trying 127.0.0.1...
* connect to 127.0.0.1 port 5000 failed: Connection refused
* Failed to connect to localhost port 5000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 5000: Connection refused

ubuntu@k8s15m1:~$ kubectl get po kube-registry-proxy-k8s15m1 --namespace=kube-system -o wide
NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE
kube-registry-proxy-k8s15m1   1/1       Running   3          1h        10.233.69.64   k8s15m1

ubuntu@k8s15m1:~$ curl -v 10.233.69.64:5000/v1/
*   Trying 10.233.69.64...
* Connected to 10.233.69.64 (10.233.69.64) port 5000 (#0)
> GET /v1/ HTTP/1.1
> Host: 10.233.69.64:5000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Tue, 14 Mar 2017 16:41:56 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host 10.233.69.64 left intact