What I have: Prometheus Operator Helm chart deployed on a single Kubernetes cluster the same which is also application cluster. So, all of the application-related pods, prometheus exporter pods, Grafana pod and Prometheus itself live within the same cluster.
What I want: I want to split the above configuration is such a way, that I have all the application-related as well as exporter pods exist in application cluster, while having Prometheus and Grafana pods deployed to separate one(in order to not have single point of failure)
Is there a way to achieve this using Prometheus Operator?
Technically, maybe, kind of. Using kubefed you can sync object state between clusters. You would need to have a mutually flat network so that the Prom pod in one cluster can talk directly to the target pods in the other.
Overall this is probably not a good idea though. Set up prom in each application cluster and then use prom-to-prom federation instead to have a single overall install.
It's not quite what you're asking, but you could run these workloads in the same cluster, just on different nodes using node selectors or taints.
It would keep the monitoring/observability workloads separate from the application workloads without having to re-architect the cluster's network or deal with federation.