I'd like to submit a SparkApplication to a Kubernetes cluster programmatically from python.
A job definition job.yaml like this
apiVersion: sparkoperator.k8s.io/v1beta1
kind: SparkApplication
metadata:
name: my-test
namespace: default
spec:
sparkVersion: "2.4.0"
type: Python
...runs without problems using kubectl apply -f job.yaml, but I cannot figure out whether and how I can use the kubernetes-client to start this job programmatically.
Does anyone know how to do this?
This is probably what you are looking for:
from __future__ import print_function
import time
import kubernetes.client
from kubernetes.client.rest import ApiException
from pprint import pprint
# Configure API key authorization: BearerToken
configuration = kubernetes.client.Configuration()
configuration.api_key['authorization'] = 'YOUR_API_KEY'
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
# configuration.api_key_prefix['authorization'] = 'Bearer'
# create an instance of the API class
api_instance = kubernetes.client.CustomObjectsApi(kubernetes.client.ApiClient(configuration))
group = 'group_example' # str | The custom resource's group name
version = 'version_example' # str | The custom resource's version
namespace = 'namespace_example' # str | The custom resource's namespace
plural = 'plural_example' # str | The custom resource's plural name. For TPRs this would be lowercase plural kind.
body = NULL # object | The JSON schema of the Resource to create.
pretty = 'pretty_example' # str | If 'true', then the output is pretty printed. (optional)
try:
api_response = api_instance.create_namespaced_custom_object(group, version, namespace, plural, body, pretty=pretty)
pprint(api_response)
except ApiException as e:
print("Exception when calling CustomObjectsApi->create_namespaced_custom_object: %s\n" % e)Here is the example mentioned, how to create third party resource on kubernetes using kubernetes python client.
https://github.com/kubernetes-client/python/blob/master/examples/create_thirdparty_resource.md
Hope this helps.