TCP connection to ActiveMQ on kubernetes keep changing

4/8/2020

I setup an scenario as following: Web Application (deployed on VM) connects to ActiveMQ (version: 5.15.10 deployed on k8s container). The kubernetes cluster contains 2 worker nodes. I found on connection dashboard of ActiveMQ that the connection changed after some minutes.

The application throws following logs:

javax.jms.JMSException: java.io.EOFException
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1960)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:219)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.EOFException
    at java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
    at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
    at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
    at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
    ... 1 more

From the ActiveMQ side, I find the logs:

2020-04-07 15:06:09,575 | WARN  | Transport Connection to: tcp://10.171.159.210:41910 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: tcp://10.171.159.210:41910 | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker

The web application has following tcp connection in openwire format to ActiveMQ:

tcp://10.171.159.210:61616?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0

Could any one has solution to keep the connection is reused? Because, when the socket connection is broken, it could cause the message is re-delivered.

EDIT My service yaml is:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
    service.kubernetes.io/ibm-load-balancer-cloud-provider-vlan: "2760930"
  creationTimestamp: "2020-02-18T06:13:01Z"
  labels:
    run: message-broker
  name: message-broker-service
  namespace: default
  resourceVersion: "9923438"
  selfLink: /api/v1/namespaces/default/services/message-broker-service
  uid: b765e641-5215-11ea-8a3b-6eead9e5000e
spec:
  clusterIP: 172.21.166.249
  externalTrafficPolicy: Cluster
  loadBalancerIP: 10.93.197.99
  ports:
  - name: port8161
    nodePort: 32577
    port: 8161
    protocol: TCP
    targetPort: 8161
  - name: port61616
    nodePort: 31958
    port: 61616
    protocol: TCP
    targetPort: 61616
  - name: port61613
    nodePort: 31200
    port: 61613
    protocol: TCP
    targetPort: 61613
  selector:
    run: message-broker
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.93.197.99
-- Minh Trần
activemq
java
jms
kubernetes

0 Answers