I am trying to set up a job scheduler (airflow) on an EKS cluster to replace a scheduler (Jenkins) we're running directly on an ec2. This job scheduler should be able to deploy pods to the EKS cluster it's running on.

However, whenever I try to deploy the pod (with a pod manifest), I get the following error message:

Error from server (Forbidden): error when creating "deployment.yaml": pods "simple-pod" is forbidden: pod does not have "kubernetes.io/config.mirror" annotation, node "ip-xx.ec2.internal" can only create mirror pods

I believe the restriction has to do with the NodeRestriction plugin on the kube-apiserver running on the EKS Control Plane.

I have looked through documentation to see if I can turn this plugin off, however it does not appear to be possible through kubectl, and only possible by modifying the kube-apiserver configuration on control plane itself.

Is it possible to turn off this plugin? Or, is it possible to label a node or pod to mark that it is not subject to this plugin? More broadly, is running a job scheduler on EKS that assigns job on the same cluster a bad design choice?

If we wanted to containerize and deploy our job scheduler, do we need to instantiate a separate EKS cluster/other service to run it on?