I'm trying to deploy a function with OpenFaas project and a kubernetes cluster running on 2 Raspberry Pi 3B+. Unfortunately, the pod that should handle the function is going to ImageInspectError state... I tried to run the function with Docker directly and which is contain in a Docker image, and everything is working fine.

I opened an issue on the OpenFaas github and the maintainer told me to ask directly the Kubernetes community to get some clues.

My first question is : What does ImageInspectError mean and where it comes from ?

And here is all the informations I have :

Expected Behaviour

Pod should run.

Current Behaviour

NAMESPACE     NAME                                 READY     STATUS              RESTARTS   AGE
kube-system   etcd-masternode                      1/1       Running             1          1d
kube-system   kube-apiserver-masternode            1/1       Running             1          1d
kube-system   kube-controller-manager-masternode   1/1       Running             1          1d
kube-system   kube-dns-7f9b64f644-x42sr            3/3       Running             3          1d
kube-system   kube-proxy-wrp6f                     1/1       Running             1          1d
kube-system   kube-proxy-x6pvq                     1/1       Running             1          1d
kube-system   kube-scheduler-masternode            1/1       Running             1          1d
kube-system   weave-net-4995q                      2/2       Running             3          1d
kube-system   weave-net-5g7pd                      2/2       Running             3          1d
openfaas-fn   figlet-7f556fcd87-wrtf4              1/1       Running             0          4h
openfaas-fn   testfaceraspi-7f6fcb5897-rs4cq       0/1       ImageInspectError   0          2h
openfaas      alertmanager-66b98dd4d4-kcsq4        1/1       Running             1          1d
openfaas      faas-netesd-5b5d6d5648-mqftl         1/1       Running             1          1d
openfaas      gateway-846f8b5686-724q8             1/1       Running             2          1d
openfaas      nats-86955fb749-7vsbm                1/1       Running             1          1d
openfaas      prometheus-6ffc57bb8f-fpk6r          1/1       Running             1          1d
openfaas      queue-worker-567bcf4d47-ngsgv        1/1       Running             2          1d

The testfaceraspi doesn't run.

Logs from the pod :

$ kubectl logs testfaceraspi-7f6fcb5897-rs4cq -n openfaas-fn
Error from server (BadRequest): container "testfaceraspi" in pod "testfaceraspi-7f6fcb5897-rs4cq" is waiting to start: ImageInspectError

Pod describe :

$ kubectl describe pod -n openfaas-fn  testfaceraspi-7f6fcb5897-rs4cq 
Name:           testfaceraspi-7f6fcb5897-rs4cq
Namespace:      openfaas-fn
Node:           workernode/10.192.79.198
Start Time:     Thu, 12 Jul 2018 11:39:05 +0200
Labels:         faas_function=testfaceraspi
                pod-template-hash=3929761453
Annotations:    prometheus.io.scrape=false
Status:         Pending
IP:             10.40.0.16
Controlled By:  ReplicaSet/testfaceraspi-7f6fcb5897
Containers:
  testfaceraspi:
    Container ID:   
    Image:          gallouche/testfaceraspi
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImageInspectError
    Ready:          False
    Restart Count:  0
    Liveness:       exec [cat /tmp/.lock] delay=3s timeout=1s period=10s #success=1 #failure=3
    Readiness:      exec [cat /tmp/.lock] delay=3s timeout=1s period=10s #success=1 #failure=3
    Environment:
      fprocess:  python3 index.py
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5qhnn (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-5qhnn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5qhnn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From                 Message
  ----     ------            ----                ----                 -------
  Warning  DNSConfigForming  2m (x1019 over 3h)  kubelet, workernode  Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch

And the event logs :

$ kubectl get events --sort-by=.metadata.creationTimestamp -n openfaas-fn
LAST SEEN   FIRST SEEN   COUNT     NAME                                              KIND         SUBOBJECT                        TYPE      REASON                  SOURCE                  MESSAGE
14m         1h           347       testfaceraspi-7f6fcb5897-rs4cq.1540db41e89d4c52   Pod                                           Warning   DNSConfigForming        kubelet, workernode     Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch
4m          1h           75        figlet-7f556fcd87-wrtf4.1540db421002b49e          Pod                                           Warning   DNSConfigForming        kubelet, workernode     Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch
10m         10m          1         testfaceraspi-7f6fcb5897-d6z78.1540df9ed8b91865   Pod                                           Normal    Scheduled               default-scheduler       Successfully assigned testfaceraspi-7f6fcb5897-d6z78 to workernode
10m         10m          1         testfaceraspi-7f6fcb5897.1540df9ed6eee11f         ReplicaSet                                    Normal    SuccessfulCreate        replicaset-controller   Created pod: testfaceraspi-7f6fcb5897-d6z78
10m         10m          1         testfaceraspi-7f6fcb5897-d6z78.1540df9eef3ef504   Pod                                           Normal    SuccessfulMountVolume   kubelet, workernode     MountVolume.SetUp succeeded for volume "default-token-5qhnn" 
4m          10m          27        testfaceraspi-7f6fcb5897-d6z78.1540df9eef5445c0   Pod                                           Warning   DNSConfigForming        kubelet, workernode     Search Line limits were exceeded, some search paths have been omitted, the applied search line is: openfaas-fn.svc.cluster.local svc.cluster.local cluster.local heig-vd.ch einet.ad.eivd.ch web.ad.eivd.ch
8m          9m           8         testfaceraspi-7f6fcb5897-d6z78.1540df9f670d0dad   Pod          spec.containers{testfaceraspi}   Warning   InspectFailed           kubelet, workernode     Failed to inspect image "gallouche/testfaceraspi": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument
9m          9m           7         testfaceraspi-7f6fcb5897-d6z78.1540df9f670fcf3e   Pod          spec.containers{testfaceraspi}   Warning   Failed                  kubelet, workernode     Error: ImageInspectError

Steps to Reproduce (for bugs)

1. Deploy OpenFaas on a 2 node k8s cluster
2. Create function with faas new testfaceraspi --lang python3-armhf

3. Add the following code in the handler.py :

   import json
   def handle(req):
       jsonl = json.loads(req)
   
       return ("Found " + str(jsonl["nbFaces"]) + " faces in OpenFaas Function on raspi !")

4. Change gateway and image in the .yml provider: name: faas gateway: http://127.0.0.1:31112

   functions:
     testfaceraspi:
       lang: python3-armhf
       handler: ./testfaceraspi
       image: gallouche/testfaceraspi

5. Run faas build -f testfacepi.yml

6. Login in DockerHub with docker login

7. Run faas push -f testfacepi.yml
8. Run faas deploy -f testfacepi.yml

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ):

  Commit: 3995a8197f1df1ecdf524844477cffa04e4690ea
  Version: 0.6.11

• Docker version ( Full output from: docker version ):

  Client:
   Version:       18.04.0-ce
   API version:   1.37
   Go version:    go1.9.4
   Git commit:    3d479c0
   Built: Tue Apr 10 18:25:24 2018
   OS/Arch:       linux/arm
   Experimental:  false
   Orchestrator:  swarm
  
   Server:
   Engine:
    Version:      18.04.0-ce
    API version:  1.37 (minimum version 1.12)
    Go version:   go1.9.4
    Git commit:   3d479c0
    Built:        Tue Apr 10 18:21:25 2018
    OS/Arch:      linux/arm
    Experimental: false

• Operating System and version (e.g. Linux, Windows, MacOS):

  Distributor ID: Raspbian
  Description:    Raspbian GNU/Linux 9.4 (stretch)
  Release:        9.4
  Codename:       stretch

Thanks by advance and tell me if you need any more informations.

Gallouche