I have a gce airflow (composer) cluster with a bunch of workers:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
airflow-redis-0 1/1 Running 0 7h
airflow-scheduler 2/2 Running 0 7h
airflow-sqlproxy 1/1 Running 0 8h
airflow-worker 50/50 Running 0 7h
composer-fluentd-daemon 1/1 Running 0 7h
composer-fluentd-daemon 1/1 Running 0 7hI also have a bunch of unique persistent NFS volumes that have data that need processing. Is there a way to dynamically mount a different NFS volume to each of the respective workers.
Alternatively, is it possible for the DockerOperator called within the worker to mount the NFS volume pertaining to its specific workload.
In theory the workflow would be: Spin up 1x worker per Dataset > Get Dataset > Run Dataset through Model > Dump results
One way to accomplish this would be to download the Dataset to the given pod that is processing it; however, these Datasets are several hundred gb per and will need to be processed many times against different models.
Eventually we plan on putting all of this data in BigTable, but I need to show a proof in concept using volumes with a few hundred gb of data before we get the green light to spin up a BigTable cluster with multiple tb of data in it.
Input appreciated. Telling me im doing it wrong with a better solution is also a viable answer.
Deployment, by definition, uses a set of identical replicas as pods (i.e. ReplicaSet). Therefore all pods of a deployment will have the PodSpec, pointing to the same volume.
Sounds like you need to write some custom logic yourself to orchestrate spinning up new workloads (i.e. Jobs) with different volumes.
You can do this by simply deploying a bash script that calls into kubectl (by default, kubectl inside a pod can work directly) in a loop. Or you can write something that uses Kubernetes API and makes some API calls to discover the new volumes, create workloads to process them (and then maybe clean up the volumes).
The workflow you describe better matches the model of a Job than a plain (long-running) Pod. You would need to create a separate job spec for each task pointing at its respective data, and if your cluster is doing other work, you would need to take care that your bulk-data-processing pods don't overwhelm the available compute resources.
Do you actually have distinct NFS volumes (server names/exported directories), or just many file trees in a single NFS volume? If the latter, another path that could work well for you is to set up a queueing system like RabbitMQ and load all of the paths into a queue there. You would then write a long-running process that serially reads a single item off the queue, does whatever required work on it, writes its result, commits the work item, and repeats (in a single thread). Then you'd scale that up using a Deployment to the desired amount of parallel processing.
In any case, your output suggests you're working directly with bare pods and have tried to scale the workers by having many parallel worker containers in a single pod. It's better to use one of the higher-level controllers (most often a Deployment) and use its replicas: control to launch multiple copies of the described pod. Among other things, this will let you spread load across multiple nodes, and will let you do a rolling update where pods get incrementally restarted, avoiding an outage if you change the underlying image or other details of the pod.