I am trying to scrape metrics for spark driver and executor using javaagent with below options. I have Prometheus in kubernetes cluster and I am running this spark application outside the kubernetes cluster.
spark.executor.extraJavaOptions=-javaagent:/opt/clkd/prometheus/jmx_prometheus_javaagent-0.3.1.jar=53700:executor_pattern.yamlbut I got below exception since both executors running on the same machine
Caused by: java.net.BindException: Address already in use .... I see many have posted the same question but I couldn't find the answer. Please let me know how can I resolve this issue.
I think that you need to switch from the pull-based monitoring, to push-based monitoring. For things, such as Spark jobs it makes more sense, as they aren't running all the time. For that you have some alternatives:
Initial answer:
You can't have 2 processes listening on the same port, so just bind Prometheus from different jobs onto the different ports. Port is the number after the jmx_prometheus_javaagent-0.3.1.jar=, and before : character - in your case it's 53700. So you can use one port for one task, and another port (maybe 53701) for 2nd task...