K
Q

Cannot push to Kubernetes registry

7/18/2018

i have a problem to push a docker image to my private registry. It starts but stops within 1% and then timeouts.

My yml files are:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: auth
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
data:
  htpasswd: |
    admin:$apr1$wBvX6bJc$CcXZK4jkLS2mxt6U3lH9p/
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry.data.efs.a
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    usage: registry-data-a
spec:
  capacity:
    storage: 100Gi
  storageClassName: value
  accessModes:
    - ReadWriteMany
  nfs:
    # TO_CHANGE: EFS AZ a endpoint
    server: eu-west-1a.fs-xxxxxx.efs.eu-west-1.amazonaws.com
    path: "/"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: registry.data.efs.a
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: value
  resources:
    requests:
      storage: 100Gi
  selector:
    matchLabels:
      usage: registry-data-a
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2.2.1
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_AUTH
          value: htpasswd
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: /auth/htpasswd
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: Registry Realm 
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
        volumeMounts:
          - mountPath: /var/lib/registry
            name: registry-data
          - mountPath: /auth
            name: auth
      volumes:
        - name: registry-data
          persistentVolumeClaim:
            claimName: registry.data.efs.a
        - name: auth
          configMap:
              name: auth
---
apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
  annotations:
    dns.alpha.kubernetes.io/internal: "registry.internal.mycompany.com"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-1:xxxxxxxxxx:certificate/3a1xx61-5xxxx-xxxx-9xxx-a4e530xxxxx
    service.beta.kubernetes.io/aws-load-balancer-instance-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443'
spec:
  selector:
    k8s-app: kube-registry
  type: LoadBalancer
  ports:
  - name: registry
    port: 443
    targetPort: 5000
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: kube-registry-node
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
spec:
  type: NodePort
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    nodePort: 30500
    protocol: TCP

I have a certificate for mycompany.com with *.mycompany.com and *.internal.mycompany.com.

So in Kubernetes everthing is running. Now when i try to push an image like this:

$ docker login registry.internal.mycompany.com
Username (admin): admin
Password:
Login Succeeded

$ docker pull busybox
$ docker tag busybox registry.internal.mycompany.com/busybox
$ docker push registry.internal.mycompany.com/busybox
The push refers to repository [registry.internal.mycompany.com/busybox]
8e9a7d50b12c: Pushing [=>                                                 ]  33.79kB/1.163MB
8e9a7d50b12c: Retrying in 15 seconds

My registry log look like this:

100.120.0.0 - - [18/Jul/2018:09:23:08 +0000] "HEAD /v2/busybox/blobs/sha256:75a0e65efd518b9bcac8a8287e5c7032bc81f8cbfbe03271fd049b81ab26119b HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-18T09:23:08Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=registry.internal.k8s-dev.hcvpc.io http.request.id=e54b3f37-14ec-4dca-9437-a3a2e1a5b120 http.request.method=POST http.request.remoteaddr="100.104.0.0:64327" http.request.uri="/v2/busybox/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=81.594324ms http.response.status=202 http.response.written=0 instance.id=db414f3f-ea65-49b8-a870-ea7657ceeaa1 version=v2.2.1
100.104.0.0 - - [18/Jul/2018:09:23:08 +0000] "POST /v2/busybox/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-18T09:23:43Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=registry.internal.k8s-dev.hcvpc.io http.request.id=719f00cc-0906-4231-8650-8f97096f47ca http.request.method=POST http.request.remoteaddr="100.104.0.0:64355" http.request.uri="/v2/busybox/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=82.222012ms http.response.status=202 http.response.written=0 instance.id=db414f3f-ea65-49b8-a870-ea7657ceeaa1 version=v2.2.1
100.104.0.0 - - [18/Jul/2018:09:23:43 +0000] "POST /v2/busybox/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-18T09:24:23Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=registry.internal.k8s-dev.hcvpc.io http.request.id=11497083-9c42-4178-8019-105fd60f70eb http.request.method=POST http.request.remoteaddr="100.104.0.0:64387" http.request.uri="/v2/busybox/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=90.719286ms http.response.status=202 http.response.written=0 instance.id=db414f3f-ea65-49b8-a870-ea7657ceeaa1 version=v2.2.1
100.104.0.0 - - [18/Jul/2018:09:24:23 +0000] "POST /v2/busybox/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.4 kernel/4.14.51-60.38.amzn1.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"

I try this now for days and have no clue. I'm pretty new to kubernetes so please don't be to harsh ;).

-- MaddEye
amazon-web-services
cluster-computing
docker-registry
kubernetes

Similar Questions

Gcloud Kubernetes connection refused to exposed service
How to connect an AWS Load Balancer to nginx ingress controller with Kubernetes deployed on EC2 with kubeadm?
How to view cluster in Kubernetes Beta under the Resources section of StackDriver?
List of all possible status / reasons in Kubernetes
container stop writing logs

0 Answers