I am having a stable/spark Helm deployment running on Kubernetes and submitting the job through Livy.
curl -X POST --data '{"className": "LogBundleConfigFetcher", "file": "http:///aliceparser_2.11-19.12.09.jar" ,"args": [""]}' -H "Content-Type: application/json" http://:8998/batches
I am able to see the driver logs in the Livy pod. But not able to see executor logs.
Is there any way I can see the executors log?
Livy API doesn't provide a way to access Spark Executor logs.
I would recommend you to look at the Grafana Loki project for easy logs collection in Kubernetes cluster.
Also Livy can be customized to collect Executor logs, but it'll be a great overhead on scale.