Kubernetes fails to pull image with unknown blob error

5/31/2018

I created a Docker image based on microsoft/dotnet-framework of a C#.NET console application built for Windows containers, then ensured I can run the image in a container locally. I successfully pushed the image to our Azure Container registry. Now I'm trying to create a deployment in our Azure Kubernetes service, but I'm getting an error:

Failed to pull image "container-registry/image:tag": rpc error: code = Unknown desc = unknown blob

I see this error on my deployment, pods, and replica sets in the Kubernetes dashboard.

We already have a secret that works with the azure-vote app, so I wouldn't think this is related to secrets, but I could be wrong.

So far, I've tried to create this deployment by pasting the following YAML into the Kubernetes dashboard Create dialog:

apiVersion:
kind: Deployment
metadata:
  name: somename
spec:
  selector:
    matchLabels:
      app: somename
      tier: backend
  replicas: 2
  template:
    metadata:
      labels:
        app: somename
        tier: backend
    spec:
      containers:
      - name: somename
        image: container-registry/image:tag
        ports:
        - containerPort: 9376

And I also tried running variations of this kubectl command:

kubectl run deploymentname --image=container-registry/image:tag

In my investigation so far, I've tried reading about different parts of k8s to understand what may be going wrong, but it's all fairly new to me. I think it may have to do with this being a Windows Server 2016 based image. A team member successfully added the azure-vote tutorial code to our AKS, so I'm wondering if there is a restriction on a single AKS service running deployments for both Windows and Linux based containers. I see by running az aks list that the AKS has an agentPoolProfile with "osType": "Linux", but I don't know if that means simply that the orchestrator is in Linux or if the containers in the pods have to be Linux based. I have found stackoverflow questions about the "unknown blob" error, and it seems the answer to this question might support my hypothesis, but I can't tell if that question is related to my questions.

Since the error has to do with failing to pull an image, I don't think this has to do with configuring a service for this deployment. Adding a service didn't change anything. I've tried rebuilding my app under the suspicion that the image was corrupted, but rebuilding and re-registering had no effect. Another thing that doesn't seem relevant that I read about is this question and answer regarding a manifest mismatch (which I don't completely understand yet).

I have not tried creating a local Kubernetes. I don't know if that's something folks typically do.

Summary of questions:

  1. What causes this unknown blob error? Does it have to do with a Windows container/Linux container mismatch?
  2. Does the agent pool profile affect all the nodes in the cluster, or just the "master" nodes?

Let me know if you need more information. Thanks.

-- Will
azure
azure-container-registry
c#
docker
kubernetes

1 Answer

6/1/2018

1. What causes this unknown blob error? Does it have to do with a Windows container/Linux container mismatch? It's because you're trying to run a Windows-based Docker container on a Linux host. It has nothing directly to do with Kubernetes or AKS. Currently AKS is in preview and supports only Linux environments. To be more precise, when you provision your AKS cluster (az aks create), all your k8s minions (worker nodes) will be Linux boxes and thus will not be able to run Windows-based containers.

2. Does the agent pool profile affect all the nodes in the cluster, or just the "master" nodes? It affects the worker nodes and is used to group them together logically so you can better manage workload distribution. In the future, when AKS supports both Linux and Windows, you will be able to i.e. create agent pools based on OS type and instruct k8s to deploy your Windows-based services only to the Windows-based hosts (agents).

-- dmusial
Source: StackOverflow