I am writing a spark job which uses kubernetes instead of yarn.
val spark = SparkSession.builder().appName("Demo").master(????).getOrCreate() So what should be my master, is it kubernetes or something else
Thanks
I should be in the format of k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
From the Doc here
URL with the format k8s://. Prefixing the master string with k8s:// will cause the Spark application to launch on the Kubernetes cluster, with the API server being contacted at api_server_url. If no HTTP protocol is specified in the URL, it defaults to https. For example, setting the master to k8s://example.com:443 is equivalent to setting it to k8s://https://example.com:443, but to connect without TLS on a different port, the master would be set to k8s://http://example.com:8080.