As explained here you can use taints and tolerations to ensure that some pods will not be scheduled on nodes with GPUs.

All nodes with GPU can be tainted like this:

kubectl taint nodes <nodename> hasgpu=true:NoSchedule

Now add the following to specs of pods - which need GPU. This will ensure that any pod which does not have this toleration will not go to an instance with GPU attached.

tolerations:
- key: "hasgpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

You can check out a detailed explanation and examples of taints and toleration in this blog

Though adding the toleration in YAML file is not as much clean and you can use an admission controller to dynamically add tolerations using an admission controller. This will add tolerations to pods which request specific resources such as GPU. You can find more details here, this solution is elegant but relatively more work.