Here is the situation.
Trying to run a Python Flask API in Kubernetes hosted in Raspberry Pi cluster, nodes are running Ubuntu 20. The API is containerized into a Docker container on the Raspberry Pi control node to account for architecture differences (ARM).
When the API and Mongo are ran outside K8s on the Raspberry Pi, just using Docker run command, the API works correctly; however, when the API is applied as a Deployment on Kubernetes the pod for the API fails with a CrashLoopBackoff and logs show 'standard_init_linux.go:211: exec user process caused "exec format error"'
Investigations show that the exec format error might be associated with problems related to building against different CPU architectures. However, having build the Docker image on a Raspberry Pi, and are successfully running the API on the architecture, I am unsure this could the source of the problem.
It has been two days and all attempts have failed. Can anyone help?
Fixed; however, something doesn't seem right.
The Kubernetes Deployment was always deployed onto the same node. I connected to that node and ran the Docker container and it wouldn't run; the "exec format error" would occur. So, it looks like it was a node specific problem.
I copied the API and Dockerfile onto the node and ran Docker build to create the image. It now runs. That does not make sense as the Docker image should have everything it needs to run.
Maybe it's because a previous image build against x86 (the development machine) remained in that nodes Docker cache/repository. Maybe the image on the node is not overwritten with newer images that have the same name and version number (the version number didn't increment). That would seem the case as the spin up time of the image on the remote node is fast suggesting the new image isn't copied on the remote node. That likely to be what it is.
I will post this anyway as it might be useful.
Edit: allow me to clarify some more, the root of this problem was ultimately because there was no shared image repository in the cluster. Images were being manually copied onto each RPI (running ARM64) from a laptop (not running ARM64) and this manual process caused the problem.
An image build on the laptop was based from a base image incompatible with ARM64; this was manually copied to all RPI's in the cluster. This caused the Exec Format error.
Building the image on the RPI pulled a base image that supported ARM64; however, this build had to be done on all RPI because there was no central repository in the cluster that Kubernetes could pull newly build ARM64 compatible images to other RPI nodes in the cluster.
Solution: a shared repository
Hope this helps.