K
Q

Question

Failed to pull image myapidemodocker.azurecr.io/apidemo:v4.0: rpc error: code = Unknown desc = unknown blob

6/17/2018

Any idea why I keep getting this annoying and unhelpful error code/description?

Failed to pull image myapidemodocker.azurecr.io/apidemo:v4.0: rpc error: code = Unknown desc = unknown blob

I thought of incorrect secret and followed this documentation from Microsoft with no success! [https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks][1].

Context:

I am using Visual Studio with Docker for Windows to create Windows Container image.
Image is pushed to Azure Container Register (ACR) and Deployed as Azure Container Instance. Unfortunately, I can't use ACI as production application because it is not connected to a private vNET. Can't use public IP for security reason but that's what is done just for poc!
Next step, Created Kubernetes cluster in Azure and trying to deploy the same image (Windows container) into Kubernetes POD but it is not working.
Let me share my yml definition and event logs

Here is my yml definition:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: apidemo
spec:
  template:
    metadata:
      labels:
        app: apidemo
    spec:
      containers:
      - name: apidemo
        image: myapidemodocker.azurecr.io/apidemo:v4.0
      imagePullSecrets:
      - name: myapidemosecret
      nodeSelector:
       beta.kubernetes.io/os: windows


Event logs:
Events:
  Type     Reason                 Age               From                               Message
  ----     ------                 ----              ----                               -------
  Normal   Scheduled              4m                default-scheduler                  Successfully assigned apidemo-57b5fc58fb-zxk86 to aks-agentp
ool-18170390-1
  Normal   SuccessfulMountVolume  4m                kubelet, aks-agentpool-18170390-1  MountVolume.SetUp succeeded for volume "default-token-gsjhl"
  Normal   SandboxChanged         2m                kubelet, aks-agentpool-18170390-1  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling                2m (x2 over 4m)   kubelet, aks-agentpool-18170390-1  pulling image "apidemodocker.azurecr.io/apidemo:v4.0"
  Warning  Failed                 20s (x2 over 2m)  kubelet, aks-agentpool-18170390-1  Failed to pull image "apidemodocker.azurecr.io/apidemo:v4
.0": [rpc error: code = Unknown desc = unknown blob, rpc error: code = Unknown desc = unknown blob]
  Warning  Failed                 20s (x2 over 2m)  kubelet, aks-agentpool-18170390-1  Error: ErrImagePull
  Normal   BackOff                10s               kubelet, aks-agentpool-18170390-1  Back-off pulling image "apidemodocker.azurecr.io/apidemo:
v4.0"
  Warning  Failed                 10s               kubelet, aks-agentpool-18170390-1  Error: ImagePullBackOff

(5) I don't understand why Kubernetes is still using /var/run/secrets/kubernetes.io/serviceaccount from default-token-gsjhl as secrete while I specified my own!

Thanks for taking time to provide feedback.

-- Prodip

azure

kubernetes

1 Answer

6/25/2018

I was able to resolve the issue. It had nothing to do with error message! The actual problem was, I was trying to use Windows Container image and Kubernetes in Azure only support Linux Container images.

This are the actions I had to do:

Configured Ubuntu (Linux Container on Windows 10)
Configured Docker to use Linux (Switch to Linux Container).
Converted ASP.NET MVC project to ASP.NET Core using Visual Studio 2017. This was a big change to support multiple platforms including Linux.
Updated the dockerfile and docker-compose project.
Created new docker image (Linux Container).
Pushed the image to Azure Container Registry.
Created a new deployment in Kubernetes with same credential. It worked!
Created a new Service to expose the app in Kubernetes. This step created an endpoint that client can use.
My Kubernetes cluster is vNET joined and all IP's are private. So, I exposed the Kubernetes endpoint (service) via Azure API Gateway. Just for the sake of demo, I allowed anonymous access to API (API Key and jwt token are must for production app).
Here is the application flow: Client App -> Azure API Gateway -> Kubernetes Endpoint(private IP) -> Kubernetes PODs -> My Linux Container

There are lots of complexities and technology specifications are changing rapidly. So, it took me lots of reading to get it right! I am sure you can do it. Try my API from Azure Kubernetes Service here-

Here are some the configurations that I used for your information-

Dockerfile:

FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
ENV ASPNETCORE_URLS=http://+:80
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "gdt.api.demo.dotnetcore.dll"]

Docker-compose:

version: '3'

services:
  gdt-api-demo:
    image: gdt.api.demo.dotnetcore
    build:
      context: .\gdt.api.demo.dotnetcore
      dockerfile: Dockerfile

Kubernetes Deployment Definition:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: gdtapidemo
spec:
  template:
    metadata:
      labels:
        app: gdtapidemo
    spec:
      containers:
      - name: gdtapidemo
        image: gdtapidemodocker.azurecr.io/gdtapidemo-ubuntu:v1.0
      imagePullSecrets:
      - name: gdtapidemosecret

Kubernetes Service Definition:

kind: Service
apiVersion: v1
metadata:
  name: gdtapidemo-service
spec:
  selector:
    app: gdtapidemo-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9200

Service as Deployed in Kubernetes

-- Prodip

Source: StackOverflow

KQ

Failed to pull image myapidemodocker.azurecr.io/apidemo:v4.0: rpc error: code = Unknown desc = unknown blob

Similar Questions

1 Answer

K
Q