My pods/containers run on a docker image which is about 4GiB in size. Pulling the image from the container registry takes about 2 mins whenever a new VM node is spun up when resources are insufficient.
That is to say, whenever a new request comes in and the Kubernetes service auto scale up a new node, it takes 2 mins+. User has to wait 2 mins to put through a request. Not ideal. I am currently using the Azure AKS to deploy my application and using their cluster autoscaler feature.
I am using a typical deployment set up with 1 fix master pod, and 3 fix worker pods. These 3 worker pods correspond to 3 different types of requests. Each time a request comes in, the worker pod will generate a K8 Job to process the request.
BIG Question is, how can I pre pull the images so that when a new node is spun up in the Kubernetes cluster, users don't have to wait so long for the new Job to be ready?
Optimally, you should re-work your Dockerfile to try and optimize the layer sizes.
In any case, you should be able to use Packer to 6 build a VM with the Docker image pre installed, then having all other instances start with that VM somehow - https://www.packer.io/docs/builders/azure.html
You'll need to rebuild the AMI everytime you upgrade the Docker image
If you are using Azure Container Registry (ACR) for storing and pulling your images you can enable teleportation that will significantly reduce your image pull time. Refer link for more information