I'm stumped on this one, i've spent the past ~3 days on it and done plenty of debugging/research attempts to no avail.

I have a docker container of Tomcat with a WAR deployed. This container works fine when started (independently or in docker-compose). However, when this container is deployed in a kubernetes cluster, Tomcat fails to deploy the WAR app. Due to the following exception:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceProxyEs' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.LogManager
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
        ... 100 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.LogManager
        at org.apache.log4j.Logger.getLogger(Logger.java:104)
        at org.elasticsearch.common.logging.log4j.Log4jESLoggerFactory.newInstance(Log4jESLoggerFactory.java:38)
        at org.elasticsearch.common.logging.ESLoggerFactory.newInstance(ESLoggerFactory.java:82)
        at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:66)
        at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:121)
        at org.elasticsearch.common.settings.Settings.<clinit>(Settings.java:63)
        at com.esri.gw.index.elasticsearch.portal.ElasticSearchClient.init(ElasticSearchClient.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1702)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1641)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570)
        ... 107 more

I confirmed that log4j jar exists in WEB-INF/lib and is loaded by adding -verbose:class to tomcat JAVA_OPTS, this is outputted in catalina startup just before the above error:

[Loaded org.elasticsearch.common.logging.DeprecationLogger from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]                           
[Loaded org.elasticsearch.common.logging.Loggers from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]                                     
[Loaded org.elasticsearch.common.Classes from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]                                             
[Loaded org.elasticsearch.common.logging.ESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar][Loaded org.elasticsearch.com
mon.logging.jdk.JdkESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]                                                   
[Loaded org.elasticsearch.common.logging.log4j.Log4jESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]                  
[Loaded org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]                  
[Loaded org.elasticsearch.common.logging.ESLogger from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]                                    
[Loaded org.apache.log4j.spi.LoggingEvent from file:/home/root/apache-tomcat-7.0.72/lib/log4j-1.2.16.jar]                                                                                    
[Loaded org.springframework.core.NestedExceptionUtils from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/spring-core-4.1.6.RELEASE.jar]                          
[Loaded java.lang.Throwable$PrintStreamOrWriter from /home/root/java/jdk1.8.0_111/jre/lib/rt.jar]                                                                                            
[Loaded java.lang.Throwable$WrappedPrintWriter from /home/root/java/jdk1.8.0_111/jre/lib/rt.jar]                                                                                             
[Loaded java.util.IdentityHashMap$KeySet from /home/root/java/jdk1.8.0_111/jre/lib/rt.jar]        

How to resolve this? And why does it occur only when container is deployed to k8s v1.9 which is running docker v1.2?

EDIT: this is not a duplicate of Copying files from host to Docker container