Been learning kubernetes in the past several weeks. I've recently built a bare-metal kubernetes cluster with (3) master nodes and (3) worker nodes (containerd runtime). Installed an another stand-alone bare-metal gitlab server with container registry enabled.

I was successful in building a simple nginx container with a custom index.html using docker build and pushed it to the registry; up until this point everything works great.

Now I wanted to create a simple pod using the image built above.
So, did the following steps. 1. Created a deploy token with read_registry access 2. Created a secret in kubernetes with username and the token as the password 3. Inserted imagePullSecrets to the deployment yaml file. 4. kubectl apply -f nginx.yaml.

Kubernetes pod status stays in ImagePullBackOff.

Failed to pull image "<gitlab-host>:5050/<user>/<project>/nginx:v1": rpc error: code = FailedPrecondition desc = failed to pull and unpack image  
"<gitlab-host>:5050/<user>/<project>/nginx:v1": failed commit on ref "unknown-sha256:4ca40a571e91ac4c425500a504490a65852ce49c1f56d7e642c0ec44d13be252": unexpected commit digest sha256:0d899af03c0398a85e36d5cd7ee9a8828e5618db255770a4a96331785ff26d9c, expected sha256:4ca40a571e91ac4c425500a504490a65852ce49c1f56d7e642c0ec44d13be252: failed precondition. 

Troubleshooting steps followed. 1. docker login from another server works. 2. docker pull works 3. In one of the worker nodes where kubernetes was scheduling the pod, I did ctr image pull which works

Did some googling but couldn't find any solutions. So, here I am as a last resort to figure this out.
Appreciate any help that I get.

My Deployment nginx.yml file

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: <gitlab-host>:5050/<username>/<project>/nginx:v1
        imagePullPolicy: IfNotPresent
        name: nginx
      imagePullSecrets:
      - name: regcred