I have a Artemis 2.11.0 broker instance deployed in our Kubernetes cluster with following configuration.
Broker.xml
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xi="http://www.w3.org/2001/XInclude" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core ">
<name>test-artemis-broker</name>
<persistence-enabled>false</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
This value was determined through a calculation.
Your system could perform 3.38 writes per millisecond
on the current journal configuration.
That translates as a sync write every 296000 nanoseconds.
Note: If you specify 0 the system will perform writes directly to the disk.
We recommend this to be 0 if you are using journalType=MAPPED and journal-datasync=false.
-->
<journal-buffer-timeout>59999</journal-buffer-timeout>
<!--
When using ASYNCIO, this will determine the writing queue depth for libaio.
-->
<journal-max-io>1</journal-max-io>
<!--
You can verify the network health of a particular NIC by specifying the <network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>296000</page-sync-timeout>
<!-- the system will enter into page mode once you hit this limit.
This is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your needs.
<global-max-size>100Mb</global-max-size>
-->
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false
as duplicate detection requires applicationProperties to be parsed on the server. -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644 for more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;writeBufferHighWaterMark=524288</acceptor>
</acceptors>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>DROP</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
<default-purge-on-no-consumers>true</default-purge-on-no-consumers>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ"/>
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue"/>
</anycast>
</address>
</addresses>
<!-- Uncomment the following if you want to use the Standard LoggingActiveMQServerPlugin pluging to log in events
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
-->
</core>
</configuration>All the queues are auto created and uses JMS client to consume messages. This is the connection URL for the clients.
Connection URL
tcp://<VIP-Name:port>?minLargeMessageSize=10485760;compressLargeMessages=trueSince the broker instance is deployed in Kubernetes, the connection go trough a VIP. I have a small pod deployed in Kubernetes that sends burst messages to a predefined queue at regular intervals. I use BYTES_TYPE messages and the size of the message is about 140Kb.
Consumer client is running outside of Kubernetes and it starts receiving messages once it is started. It receives 1500 to 2000 messages and completely stops after that. Even restarting the consumer does not help. I don't see any exceptions or errors in the consumer client as well as on the broker logs (I've enabled debug logs as well).
The same broker configuration running on a physical host works without any issues (consumer client, producer client and broker on 3 different physical hosts).
Only notable difference between two setup is the network latency. Network round trip for the Kubernetes VIP is about 29ms and for the physical host broker is about 0.14ms.
I think this network delay is triggering flow control or something similar at the broker leads to this issue.
I went through the documentation of Artemis but was not able to find anything related.
Anyone has any idea to debug this? Did I miss some broker configuration?
Any help is greatly appreciated!
PS: I can provide Java code example as well if needed.
Thread Dump
When the message receiving is stopped in the consumer.
31631:
2020-06-11 23:11:14
Full thread dump OpenJDK 64-Bit Server VM (11.0.5+10-LTS mixed mode):
Threads class SMR info:
_java_thread_list=0x00007f836b606ff0, length=17, elements={
0x00007f838c008800, 0x00007f838c810000, 0x00007f838b84e800, 0x00007f838b85a800,
0x00007f838b85d800, 0x00007f838c00b800, 0x00007f836b80a000, 0x00007f838c85b800,
0x00007f838c061000, 0x00007f838d2e5800, 0x00007f838c0d1000, 0x00007f838c0d2000,
0x00007f838baf8000, 0x00007f838c888000, 0x00007f835c0ec800, 0x00007f835b940000,
0x00007f838c0f2800
}
"main" #1 prio=5 os_prio=31 cpu=2302.27ms elapsed=689.47s tid=0x00007f838c008800 nid=0x2703 in Object.wait() [0x00007000084a8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.5/Native Method)
- waiting on <no object reference available>
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:268)
- waiting to re-lock in wait() <0x000000040045f1f0> (a org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl)
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:395)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.getMessage(ActiveMQMessageConsumer.java:213)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:134)
at com.apple.carnival.artemis.ArtemisReceiver.startReceiving(ArtemisReceiver.java:52)
at com.apple.carnival.artemis.ArtemisReceiver.main(ArtemisReceiver.java:38)
"Reference Handler" #2 daemon prio=10 os_prio=31 cpu=0.34ms elapsed=689.45s tid=0x00007f838c810000 nid=0x3203 waiting on condition [0x0000700008bbd000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.5/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base@11.0.5/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.5/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=31 cpu=0.24ms elapsed=689.45s tid=0x00007f838b84e800 nid=0x3403 in Object.wait() [0x0000700008cc0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.5/Native Method)
- waiting on <0x0000000400004150> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.5/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000400004150> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.5/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.5/Finalizer.java:170)
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=0.21ms elapsed=689.43s tid=0x00007f838b85a800 nid=0x3903 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 cpu=2689.07ms elapsed=689.43s tid=0x00007f838b85d800 nid=0x3f03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread0" #13 daemon prio=9 os_prio=31 cpu=357.45ms elapsed=689.43s tid=0x00007f838c00b800 nid=0x3c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #17 daemon prio=9 os_prio=31 cpu=29.48ms elapsed=689.43s tid=0x00007f836b80a000 nid=0x5603 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #18 daemon prio=9 os_prio=31 cpu=0.08ms elapsed=689.41s tid=0x00007f838c85b800 nid=0xa603 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #19 daemon prio=8 os_prio=31 cpu=1.07ms elapsed=689.41s tid=0x00007f838c061000 nid=0xa203 in Object.wait() [0x00007000095de000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.5/Native Method)
- waiting on <no object reference available>
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.5/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000400002b48> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@11.0.5/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
at jdk.internal.misc.InnocuousThread.run(java.base@11.0.5/InnocuousThread.java:134)
"Thread-0 (ActiveMQ-client-netty-threads)" #31 daemon prio=5 os_prio=31 cpu=19934.95ms elapsed=688.81s tid=0x00007f838d2e5800 nid=0x5f07 runnable [0x00007000098e7000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native Method)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:147)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:139)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:208)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
"Thread-0 (ActiveMQ-client-global-scheduled-threads)" #32 daemon prio=5 os_prio=31 cpu=9.13ms elapsed=688.75s tid=0x00007f838c0d1000 nid=0x6607 waiting on condition [0x00007000099ea000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
- parking to wait for <0x00000004000002f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.5/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.5/AbstractQueuedSynchronizer.java:2123)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.5/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
"Thread-1 (ActiveMQ-client-global-scheduled-threads)" #33 daemon prio=5 os_prio=31 cpu=8.42ms elapsed=688.74s tid=0x00007f838c0d2000 nid=0x9507 waiting on condition [0x0000700009aed000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
- parking to wait for <0x00000004000002f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.5/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.5/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
"Thread-0 (ActiveMQ-client-global-threads)" #34 daemon prio=5 os_prio=31 cpu=361.89ms elapsed=688.69s tid=0x00007f838baf8000 nid=0x9c07 waiting on condition [0x0000700009bf0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
- parking to wait for <0x0000000400001340> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.5/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.5/AbstractQueuedSynchronizer.java:2123)
at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.5/LinkedBlockingQueue.java:458)
at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.5/ThreadPoolExecutor.java:1053)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
"Thread-2 (ActiveMQ-client-global-scheduled-threads)" #35 daemon prio=5 os_prio=31 cpu=10.34ms elapsed=658.73s tid=0x00007f838c888000 nid=0x5d17 waiting on condition [0x00007000092d5000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
- parking to wait for <0x00000004000002f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.5/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.5/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
"Thread-3 (ActiveMQ-client-global-scheduled-threads)" #36 daemon prio=5 os_prio=31 cpu=0.35ms elapsed=628.72s tid=0x00007f835c0ec800 nid=0x5a13 waiting on condition [0x00007000096e1000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
- parking to wait for <0x00000004000002f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.5/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.5/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
"Thread-4 (ActiveMQ-client-global-scheduled-threads)" #37 daemon prio=5 os_prio=31 cpu=0.43ms elapsed=598.72s tid=0x00007f835b940000 nid=0x5b0b waiting on condition [0x00007000097e4000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
- parking to wait for <0x00000004000002f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.5/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.5/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.5/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
"Attach Listener" #38 daemon prio=9 os_prio=31 cpu=1.44ms elapsed=16.55s tid=0x00007f838c0f2800 nid=0x8c13 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" os_prio=31 cpu=67.67ms elapsed=689.45s tid=0x00007f838d199000 nid=0x3103 runnable
"GC Thread#0" os_prio=31 cpu=25.99ms elapsed=689.46s tid=0x00007f838c00b000 nid=0x5103 runnable
"GC Thread#1" os_prio=31 cpu=23.67ms elapsed=687.48s tid=0x00007f838ba9a000 nid=0x6b07 runnable
"GC Thread#2" os_prio=31 cpu=24.96ms elapsed=687.48s tid=0x00007f838d358800 nid=0x9207 runnable
"GC Thread#3" os_prio=31 cpu=23.64ms elapsed=687.48s tid=0x00007f838c0e8800 nid=0x9307 runnable
"GC Thread#4" os_prio=31 cpu=24.06ms elapsed=687.48s tid=0x00007f838c0f4000 nid=0x9407 runnable
"GC Thread#5" os_prio=31 cpu=24.32ms elapsed=687.48s tid=0x00007f838c0f4800 nid=0x6907 runnable
"GC Thread#6" os_prio=31 cpu=23.72ms elapsed=687.48s tid=0x00007f838c0f5800 nid=0x9107 runnable
"GC Thread#7" os_prio=31 cpu=23.95ms elapsed=687.48s tid=0x00007f838c0f6000 nid=0x9b07 runnable
"GC Thread#8" os_prio=31 cpu=24.06ms elapsed=687.48s tid=0x00007f838c889800 nid=0x6107 runnable
"GC Thread#9" os_prio=31 cpu=24.33ms elapsed=687.48s tid=0x00007f838c88a800 nid=0x9707 runnable
"GC Thread#10" os_prio=31 cpu=23.21ms elapsed=687.48s tid=0x00007f838c87e800 nid=0x6e03 runnable
"GC Thread#11" os_prio=31 cpu=24.65ms elapsed=687.48s tid=0x00007f838c87f000 nid=0x9003 runnable
"GC Thread#12" os_prio=31 cpu=23.26ms elapsed=687.48s tid=0x00007f838c880000 nid=0x7003 runnable
"G1 Main Marker" os_prio=31 cpu=0.13ms elapsed=689.46s tid=0x00007f838b84a000 nid=0x2c03 runnable
"G1 Conc#0" os_prio=31 cpu=0.03ms elapsed=689.46s tid=0x00007f838d008800 nid=0x4d03 runnable
"G1 Refine#0" os_prio=31 cpu=0.20ms elapsed=689.46s tid=0x00007f838d13d000 nid=0x2e03 runnable
"G1 Young RemSet Sampling" os_prio=31 cpu=169.05ms elapsed=689.46s tid=0x00007f838d13e000 nid=0x2f03 runnable
"VM Periodic Task Thread" os_prio=31 cpu=357.44ms elapsed=689.41s tid=0x00007f838b909000 nid=0xa403 waiting on condition
JNI global refs: 35, weak refs: 0Update 1
Above scenario is when I ran my producer in the same Kubernetes cluster to avoid any network latency. The consumer is running outside of the cluster.
Then I stopped the producer in Kubernetes cluster and started a one outside. After sending ~24 messages to the broker, I got following error.
Jun 11, 2020 11:46:18 PM org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl actualAcquire
WARN: AMQ212054: Destination address=test.controller.queue is blocked. If the system is configured to block make sure you consume messages on this configuration.