I'm current using Apache Airflow on Kubernetes (aws EKS) to schedule batch processing jobs. For large infrequent jobs I'm using EKS Fargate.
According to the docs here:
When provisioned, each pod running on Fargate receives 20 GB of container image layer storage. Pod storage is ephemeral.
I am running workloads that require larger ephemeral storage than 20 GBs. Reading the docs here on ECS Fargate make me think it's possible (at least for ECS) to modify the initial ephemeral storage:
The minimum supported value is 21 GiB and the maximum supported value is 200 GiB.
Lastly, I'm currently using KubernetesPodOperator in airflow, which allows me to add annotations, labels, tags, resource limits...etc. when launching Fargate pods.
Is it possible to launch Fargate pods on EKS with larger default ephemeral storage? If so how?
It is not possible to use the Fargate extended ephemeral local storage option with EKS (it is only possible with ECS for now). An alternative you can explore (if you haven't done so already) is to mount an EFS volume to the Fargate pods. While EFS may have different performance characteristics than local ephemeral storage, EFS would allow you to mount the same volume to multiple pods (if that architecture is supported by and useful for your application).