Kubernetes secrets and spring boot configuration

6/17/2018

Our service is running in kubernetes cluster. I'm trying to make our service to be secured by SSL.

For that purpose I added to application.properties:

security.require-ssl=true 
server.ssl.key-store-type=JKS
server.ssl.key-store=serviceCertificates.jks
server.ssl.key-store-password=${KEYSTORE_PASSWORD}
server.ssl.key-alias=certificate

The keystore password I want to take from kubernetes secret, that is defined in the cluster.
When the service starts running I get an error Password verification failed:

"org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-8080]]\n\tat org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)\n\tat org.apache.catalina.core.StandardService.addConnector(StandardService.java:225)\n\tat org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:256)\n\tat org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:198)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.startWebServer(ServletWebServerApplicationContext.java:300)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162)\n\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)\n\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)\n\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:327)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)\n\tat com.ibm.securityservices.cryptoutils.Application.main(Application.java:9)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:50)\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)\nCaused by: org.apache.catalina.LifecycleException: Protocol handler start failed\n\tat org.apache.catalina.connector.Connector.startInternal(Connector.java:1020)\n\tat org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)\n\t... 21 common frames omitted\nCaused by: java.lang.IllegalArgumentException: Keystore was tampered with, or password was incorrect\n\tat org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:116)\n\tat org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:87)\n\tat org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:225)\n\tat org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1150)\n\tat org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:591)\n\tat org.apache.catalina.connector.Connector.startInternal(Connector.java:1018)\n\t... 22 common frames omitted\nCaused by: java.io.IOException: Keystore was tampered with, or password was incorrect\n\tat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)\n\tat sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)\n\tat sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)\n\tat sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)\n\tat java.security.KeyStore.load(KeyStore.java:1445)\n\tat org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:139)\n\tat org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:204)\n\tat org.apache.tomcat.util.net.jsse.JSSEUtil.getKeyManagers(JSSEUtil.java:184)\n\tat org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114)\n\t... 27 common frames omitted\nCaused by: java.security.UnrecoverableKeyException: Password verification failed\n\tat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)\n\t... 35 common frames omitted\n"}

My investigation:
1. If I print in the code

    System.out.println("KEYSTORE_PASSWORD: "+ System.getenv("KEYSTORE_PASSWORD"));   

I see its correct value.
2. If I set hard coded constant password value in application properties, it works, service is up and running.

So I guess that the problem is to set secret value to application property.
Your help and advice will be appreciated

-- Tantre
kubernetes
kubernetes-secrets
kubernetes-security
spring
spring-boot

1 Answer

6/17/2018

I think there is typo or hidden character in your secret descriptor. You can exec into the pod , verify the system property and also try decrypting the password using command line tools.

-- Raj
Source: StackOverflow