I have a Kubernetes cluster with multiple namespaces. A namespace per each client, that's how we separate clients' data.
I want to run a single airflow cluster containing DAGs per client. But is there a way to use the Kubernetes executor so that it will execute each task on a different namespace based on some parameter? I noticed the Kubernetes executor has only one namespace parameter.
Kubernetes does not support multiple namespaces for the --namespace argument currently. More info about this is here and here.
I think a single operation across multiple namespaces will be complex, and may produce inconsistent results and race conditions.
kubectl get --all-namespaces gets resources from all namespaces.
An operation can be repeatedly run in each namespace serially in a programmatic way as follows:
kubectl get ns -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -n 1 -I % kubectl get pods -n=%
kubectl get ns -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -n 1 -I % kubectl get all -n=%