I'm a newbie in kubernetes space, sorry if I'm missing anything obvious here.
I have Prometheus running in GKE and it has to scrape metrics exposed on an endpoint mounted in GCE, the host in GCE is behind a VPN, I'm not sure if this is an issue since both are on Google cloud. What can I do here to ensure that prometheus in kubernetes can connect to the host in GCE and scrape metrics form them.
Edit: added config and error message
Prometheus scrape_config
- job_name: 'cassandra-metrics'
static_configs:
- targets:
- <ip>:<port>error when trying to scrape
net_conntrack_dialer_conn_failed_total{dialer_name="cassandra-metrics",reason="timeout"} 4675Remember that the GCP default network is global, therefore it doesn't matter in which zone you deploy your resources, they will always be able to reach each other using the internal IPs or external IPs (keep in mind that if you are using external IPs to communicate you will be charged by ingress/egress traffic and also setup the necessary firewall rules).
If you are using different VPCs or networks,you need to setup VPC Peering, this will allow communication between both VPCs, for example, you have your GCE instance in one VPC and your GKE cluster in other.
I tried to replicate your scenario, a GKE cluster and a compute instance deployed in the default network, applied a firewall rule to allow ingress traffic to all instances in the network (no network tags used).
I did a deploy of busybox using this yaml file in GKE, logged in into one pod, kubectl get pods and then using kubectl exec -ti $podname sh, finally used these tools to test the connection: traceroute and ping.
The connection was successful between both resources, notice that instead of using the UDP protocol for traceroute I used the option -I which stands for "use ICMP ECHO for probes".
The instance being behind a VPN (Cloud VPN, Dedicated interconnect or direct peering) doesn't affect the fact that the GKE cluster can't reach it, unless like I mentioned before, both resources are on separate/different networks.