I have an airflow DAG where some of the tasks need to be horizontally scaled when they are kicked in depending upon their memory usage.

From the definition of HPA,

The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics).

HPA usually scales deployments, replica-sets, replication controllers. In Airflow, however, we have KubernetesPodOperator which spawns a pod. This spawned pod is managed by Airflow itself where it takes care of retries on fail-over etc.

The requirement now is:

If a pod has reached x% of memory or cpu, I would like to run another instance of the same pod (task). Can we horizontally scale the tasks? If so, how?

How do we deal with situations, in general, where a task requires more memory than what can fit in one Kubernetes node?