Cannot push to Kubernetes registry

7/18/2018

i have a problem to push a docker image to my private registry. It starts but stops within 1% and then timeouts.

My yml files are:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: auth
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
data:
  htpasswd: |
    admin:$apr1$wBvX6bJc$CcXZK4jkLS2mxt6U3lH9p/
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry.data.efs.a
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    usage: registry-data-a
spec:
  capacity:
    storage: 100Gi
  storageClassName: value
  accessModes:
    - ReadWriteMany
  nfs:
    # TO_CHANGE: EFS AZ a endpoint
    server: eu-west-1a.fs-xxxxxx.efs.eu-west-1.amazonaws.com
    path: "/"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: registry.data.efs.a
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: value
  resources:
    requests:
      storage: 100Gi
  selector:
    matchLabels:
      usage: registry-data-a
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2.2.1
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_AUTH
          value: htpasswd
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: /auth/htpasswd
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: Registry Realm 
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
        volumeMounts:
          - mountPath: /var/lib/registry
            name: registry-data
          - mountPath: /auth
            name: auth
      volumes:
        - name: registry-data
          persistentVolumeClaim:
            claimName: registry.data.efs.a
        - name: auth
          configMap:
              name: auth
---
apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
  annotations:
    dns.alpha.kubernetes.io/internal: "registry.internal.mycompany.com"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-1:xxxxxxxxxx:certificate/3a1xx61-5xxxx-xxxx-9xxx-a4e530xxxxx
    service.beta.kubernetes.io/aws-load-balancer-instance-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443'
spec:
  selector:
    k8s-app: kube-registry
  type: LoadBalancer
  ports:
  - name: registry
    port: 443
    targetPort: 5000
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: kube-registry-node
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
spec:
  type: NodePort
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    nodePort: 30500
    protocol: TCP

I have a certificate for mycompany.com with *.mycompany.com and *.internal.mycompany.com.

So in Kubernetes everthing is running. Now when i try to push an image like this:

$ docker login registry.internal.mycompany.com
Username (admin): admin
Password:
Login Succeeded

$ docker pull busybox
$ docker tag busybox registry.internal.mycompany.com/busybox
$ docker push registry.internal.mycompany.com/busybox
The push refers to repository [registry.internal.mycompany.com/busybox]
8e9a7d50b12c: Pushing [=>                                                 ]  33.79kB/1.163MB
8e9a7d50b12c: Retrying in 15 seconds

My registry log look like this:

100.120.0.0 - - [18/Jul/2018:09:23:08 +0000] "HEAD /v2/busybox/blobs/sha256:75a0e65efd518b9bcac8a8287e5c7032bc81f8cbfbe03271fd049b81ab26119b HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-18T09:23:08Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=registry.internal.k8s-dev.hcvpc.io http.request.id=e54b3f37-14ec-4dca-9437-a3a2e1a5b120 http.request.method=POST http.request.remoteaddr="100.104.0.0:64327" http.request.uri="/v2/busybox/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=81.594324ms http.response.status=202 http.response.written=0 instance.id=db414f3f-ea65-49b8-a870-ea7657ceeaa1 version=v2.2.1
100.104.0.0 - - [18/Jul/2018:09:23:08 +0000] "POST /v2/busybox/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-18T09:23:43Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=registry.internal.k8s-dev.hcvpc.io http.request.id=719f00cc-0906-4231-8650-8f97096f47ca http.request.method=POST http.request.remoteaddr="100.104.0.0:64355" http.request.uri="/v2/busybox/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=82.222012ms http.response.status=202 http.response.written=0 instance.id=db414f3f-ea65-49b8-a870-ea7657ceeaa1 version=v2.2.1
100.104.0.0 - - [18/Jul/2018:09:23:43 +0000] "POST /v2/busybox/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-18T09:24:23Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=registry.internal.k8s-dev.hcvpc.io http.request.id=11497083-9c42-4178-8019-105fd60f70eb http.request.method=POST http.request.remoteaddr="100.104.0.0:64387" http.request.uri="/v2/busybox/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=90.719286ms http.response.status=202 http.response.written=0 instance.id=db414f3f-ea65-49b8-a870-ea7657ceeaa1 version=v2.2.1
100.104.0.0 - - [18/Jul/2018:09:24:23 +0000] "POST /v2/busybox/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"

I try this now for days and have no clue. I'm pretty new to kubernetes so please don't be to harsh ;).

-- MaddEye
amazon-web-services
cluster-computing
docker-registry
kubernetes

0 Answers