I'm trying to use a library that uses uses Twitter's finagle. It works seamlessly on the driver side, but it gets stuck on the executors.

Some versions of finagle have this bug where it gets stuck if it encounters a link cycle while traversing directories. It seems the library checks all the directories on the working directory, or at least all of them on the classpath.

The problem happens just on the executor side because Spark is copying the jar and files specified with -Dspark.files to the root folder (/), which also contains directories like /sys or /proc, which have cycles.

Is there a way to tell Spark to use another directory for the working directory? Or maybe I should modify my Docker image? currently it's a simple centos image with Java installed. The user is just root, but Spark is using /, not the home of the root user (/root).