We have applications running in EKS, which complain on "Address already in use" while attempting to connect with Jconsole. When we tried to run just java -version it fails on the same error. See exception below:
/home/app # java -version
Error: JMX connector server communication error: service:jmx:rmi://qa-my-app-av22b3dcbc-4322t:9110
sun.management.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 9110; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:800)
at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:468)
at sun.management.Agent.startAgent(Agent.java:262)
at sun.management.Agent.startAgent(Agent.java:452)
at java.lang.System.startSNMPAgent(Native Method)
at java.lang.Thread.completeInitialization(Thread.java:168)
at java.lang.J9VMInternals.completeInitialization(J9VMInternals.java:74)
Caused by: java.rmi.server.ExportException: Port already in use: 9110; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346)
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254)
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:237)
at sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:199)
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:146)
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:122)
at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:404)
at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:796)
... 6 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:390)
at java.net.ServerSocket.<init>(ServerSocket.java:252)
at java.net.ServerSocket.<init>(ServerSocket.java:143)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createServerSocket(RMIDirectSocketFactory.java:45)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createServerSocket(RMIMasterSocketFactory.java:345)
at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
... 15 more
Exception in thread "main" java/lang/RuntimeException: sun.management.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 9110; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at sun/management/Agent.error (Agent.java:526)
at sun/management/Agent.startAgent (Agent.java:269)
at sun/management/Agent.startAgent (Agent.java:452)
at java/lang/System.startSNMPAgent (NativeMethod:4294967295)
at java/lang/Thread.completeInitialization (Thread.java:168)
at java/lang/J9VMInternals.completeInitialization (J9VMInternals.java:74)JAVA_TOOL_OPTIONS env variable value:
-Dspring.config.location=/home/app/config/application.properties -Dlogging.config=/home/app/config/logback.xml -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9110 -Dcom.sun.management.jmxremote.rmi.port=9110 -Djava.rmi.server.hostname=127.0.0.1 -javaagent:/home/app/newrelic/newrelic.jar -Dnewrelic.config.file=/home/app/newrelic/config/newrelic.ymlThe image entrypoint is simply
ENTRYPOINT ["java", "-jar", "/home/app/application.jar"]
Seems like there's another process on that port. If it is not an important one, kill the process bia fuser, for example:
sudo fuser -k 9110/tcpYou could also check what's listening there before killing it:
lsof -i :9110When the port is clean, try again. Another option is to change the rmi port number to something different than 9110, as jmx doesn't require specific port numbers to work (don't forget changing the port on the endpoint as well).