I'm using Sails.js for an API which I deploy from a Dockerfile in a Google Cloud kubernetes cluster and scale the workload with 3-5 pods. The API provides endpoints to upload single image files and bigger zip files which I directly extract on the current API pod/instance.

Both, single image files and the extracted archive content (100-1000 files with all together 15-85mb of content), I have to upload to various Storage buckets. This is where redis kue comes into play. To make sure the API is not blocking the request for the uploads for too long, I create delayed kue jobs to move all the uploaded files and folders to storage buckets or chain jobs and create thumbnails with the help of ImageMagick first.

All this can take some time, depending on the current workload of the cluster, sometimes more and sometimes less.

All this works pretty fine with one single instance but within a cluster, it's a different story. Since the kubernetes instance for the API can change from request to request, the uploads can land on instance A, but the job for the files is being processed and handled by instance B (The worker, as well as the API itself, are running on the same instance!) which might won't have the uploads available which leads into a failed job.

It takes time for Google to keep the pods in sync and to spread the uploads to all the other pods.

What I have tried is the following:

Since the name of the current pod is available via env variable HOSTNAME, I'm storing the HOSTNAME with all kue jobs and check within the worker if the HOSTNAME from the jobs matches with the HOSTNAME of the current environment and only allow to process the jobs if both HOSTNAMEs are matching.

Uploads need to be available ASAP; why I can't add a job delay of a few minutes and hope that by the time the job is going to be processed, Google has synchronized its pods.

Pending jobs which don't match the HOSTNAME, I push back to the queue and add delay to it.

What I want to have is a queue which doesn't have to take care of hostnames and conditional checks to successfully process its jobs in a cluster like mine.