I'm working on a Docker-based project. The project code is hosted in a private Gitlab installation, git.example.com
. With it, the Docker private registry shipped with Gitlab is deployed, registry.example.com
.
The project has a CI setup which ends up building Docker images and pushing to the registry, this part works as expected. As Gitlab+Docker registry does not yet support multiple images related to the same Git repo, I'm using the tags workaround which specifies an image as:
registry.example.com/group/my.project:web
registry.example.com/group/my.project:app
I've created a user and attached it to the projects, logged in via it locally and tried to pull above images, that works as expected.
I've added the ImageStream
block as so:
apiVersion: v1
kind: ImageStream
metadata:
name: web
spec:
tags:
-
from:
kind: DockerImage
name: registry.example.com/group/my.project:web
name: latest
This adds the image in the Images section, but it cannot pull it Openshift doesn't have access to the Docker Registry yet. I add a new Docker secret as described here and am now able to see image metadata in Openshift, everything looks as expected.
But, if I add a deployment config, like so:
apiVersion: v1
kind: DeploymentConfig
metadata:
creationTimestamp: null
labels:
service: web
name: web
spec:
replicas: 1
selector:
service: web
strategy:
resources: { }
template:
metadata:
creationTimestamp: null
labels:
service: web
spec:
containers:
-
name: web
ports:
-
containerPort: 80
resources: { }
restartPolicy: Always
test: false
triggers:
-
type: ConfigChange
-
type: ImageChange
imageChangeParams:
automatic: true
containerNames:
- web
from:
kind: ImageStreamTag
name: 'web:latest'
status: { }
I keep getting error:
Failed to pull image "registry.example.com/group/my.project@sha256:3333022641e571d7e4dcae2953d35be8cdf9416b13967b99537c4e8f150f74e4": manifest unknown: manifest unknown
in the Events tab of the pod created. This basically kills my plan to deploy prebuilt images to Openshift.
I know about Docker 1.9 -> 1.10 incompatibility, but this is Openshift 1.4.1, images were pushed with Docker 1.13 so it shouldn't be a problem.
How do I even start debugging this, is there a way to access any sort of log which would explain what's going on? Why is ImageStream able to find everything it needs (and access my registry), but not the DeploymentConfig?
To answer my own question: it seems Docker's Distribution (registry daemon) has a bug which manifests itself in quite a weird way.
Basically, the problem is:
SHA:1234
(example, of course)SHA:ABCD
, you can reproduce this by pushing and then pulling right away, the digests are supposed to be identical both times, as explained in the linkI've since moved my Gitlab instance to another machine (where it's behind Nginx) and it works without a problem.