K
Q

Authenticating to any Apache Spark web UI with LDAP

12/14/2020

I am looking for a way to introduce LDAP authentication to my Apache Spark Web UIs. This includes both the main Spark Web UI and the Spark history server. The Spark instances are running inside a Kubernetes cluster and are launched both via the Spark Operator and in Standalone mode.

In the official documentation, it says the following:

"Enabling authentication for the Web UIs is done using javax servlet filters. You will need a filter that implements the authentication method you want to deploy. Spark does not provide any built-in authentication filters."

When searching online, I cannot find any such filters which will enable authentication through LDAP.

My questions are:

  1. Are there any such filters implemented that is open source and freely available?
  2. Is there another solution to my problem which will enable this feature?
-- toerq
apache-spark
java
kubernetes
ldap

Similar Questions

Running java app in kubernetes container is not working
Google Endpoints + GKE + GRPC
benefits of running k8s pods in non default namespace
Kubedb operator issues
is that possible to add more "https endpoints" for the workers kubeconfig?

1 Answer

12/16/2020
  1. Add a Java Servlet filter. I didn't find a public available ldap filter, however here is a PAM Filter, we can use https://github.com/marccarre/pam-servlet-filter. Now, we can configure PAM to use SSSD which uses LDAP.

Hint: Use thin client and use libpam4j-1.11.jar (not the official referenced version (1.9) since i could not get it work with 1.9) and jna-4.3.0.jar. This was a working setup for me. Add all of these 3 jars to SPARK_DIR/jars/

  1. On all spark nodes execute
yum install sssd authconfig -y
authconfig --update --enablesssd --enablesssdauth

  1. Add / Edit /etc/sssd/sssd.conf. Google how to configure this file. Basically you can configure here variables like ldap_uri, ldap_search_base and ldap_access_filter

  2. Add following Variables in spark-defaults.conf:

spark.ui.filters                    com.carmatechnologies.servlet.PamAuthFilter
spark.com.carmatechnologies.servlet.PamAuthFilter.param.realm spark
spark.com.carmatechnologies.servlet.PamAuthFilter.param.service system-auth
  1. Start sssd
sssd -i -d 3
  1. Start Spark UIs

Authentication should work now.

-- Natan
Source: StackOverflow