I'm try to run 2 Spring Boot applications within the same Pod (essentially one is a reverse proxy for the other one - a small implementation of the sidecar pattern) and I've that one of the containers can't start. In fact, it crashes with the following error:
Starting the Java application using /opt/jboss/container/java/run/run-java.sh ...
INFO exec java -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -javaagent:/usr/share/java/prometheus-jmx-exporter/jmx_prometheus_javaagent.jar=9779:/opt/jboss/container/prometheus/etc/jmx-exporter-config.yaml -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/app-backend-1.0.0-SNAPSHOT.jar
Could not start Jolokia agent: java.lang.IllegalStateException: Cannot open keystore for https communication: java.net.BindException: Address already in use
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:555)
at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
at jdk.httpserver/sun.net.httpserver.ServerImpl.bind(ServerImpl.java:134)
at jdk.httpserver/sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:31)
... 6 more
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f3cf9ef7e91, pid=1, tid=154
#
# JRE version: OpenJDK Runtime Environment 21.9 (17.0.1+12) (build 17.0.1+12-LTS)
# Java VM: OpenJDK 64-Bit Server VM 21.9 (17.0.1+12-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
# Problematic frame:
# C [libc.so.6+0x21e91] abort+0x203
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /deployments/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
# https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-17-openjdk
#
I've the impression the issue is caused by the way the plugin I'm using builds the final image, but I'm not sure. I suspect that both the containers use the same port (since the images are built in the same way) for the Jolokia and Prometheus JMX agents, but I didn't find a way to disable them.
Does somebody have some ideas?
when running 2 containers in one pod, the containers recognize the pod as their localhost environment. as a result, if both containers try to bind the same port, they will interfere each other.
try to configure one of the containers to bind another port. this should resolve the error above. :)
edit: I checked the provided link to the github issue you opened for the plugin. jkube and spring boot docs both list several different ways to provide build- and runtime configs. I don't think this is a bug but a configuration issue that can be solved with the options the framework and plugin provides.
I propose you dig the docs further to check how to configure the ports for the processes that are started in both containers via springs application properties, ENVs or something like that. Afterwards, you can check how to provide these values during build/runtime using jkube generated manifests.
docs I've found:
In the end it was effectively a bug in JKube. See https://github.com/eclipse/jkube/issues/1294#event-6132755248