In my k8s spark application, I would like to change the log4j log LEVEL in executor pods.

In log4j properties file, I have set rootLogger to WARN, but still in executors pods I can see it is parsing to INFO.

log4j.properties:

log4j.rootLogger=WARN,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d %-5p [%t] %c{2}(%L): %m%n

spark-submit:

spark-submit  \
--master k8s://..  \
--deploy-mode client \
--conf spark.driver.extraJavaOptions='-Dlog4j.debug=true -Dlog4j.configuration=file:///opt/log4j.properties'\
--conf spark.executor.extraJavaOptions='-Dlog4j.debug=true -Dlog4j.configuration=file:///opt/log4j.properties'\
--class org.log4jTestRunner \
logger-test-1.0.jar

Driver logs:

log4j: Using URL [file:/opt/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/opt/log4j.properties
log4j: Parsing for [root] with value=[WARN,console].
log4j: Level token is [WARN].
log4j: Category root set to WARN
log4j: Parsing appender named "console".
log4j: Parsing layout options for "console".
log4j: Setting property [conversionPattern] to [%d %-5p [%t] %c{2}(%L): %m%n].
log4j: End of parsing for "console".
log4j: Setting property [target] to [System.err].
log4j: Parsed "console" options.

Executor logs:

log4j: Using URL [file:/opt/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/opt/log4j.properties
log4j: Parsing for [root] with value=[INFO,console].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named ""console"".
log4j: Parsing layout options for ""console"".
log4j: Setting property [conversionPattern] to [%d %-5p [%t] %c{2}(%L): %m%n].
log4j: End of parsing for ""console"".
log4j: Setting property [target] to [System.err].
log4j: Parsed ""console"" options.

I can see in Driver it parsing correctly and log4j log level is respected. Not sure, why in executors it is working differently.

I am using k8s with spark 3.x version.

Thank you in advance.