K
Q

WSO2 ESB unable to reconnect to TIBCO JMS servers after switching from home TIBCO EMS server to secondary TIBCO EMS server

12/21/2020

My WSO2 ESB is containerized in a docker image and driven through Kubernetes. When my ESB pods start up, my JMS server connections are OK and JMS message sending and consumption are OK. The 2 URLs and ports of TIBCO EMS servers are correctly configured as such in the axis2.xml file :

<transportReceiver name="local" 
class="org.wso2.carbon.core.transports.local.CarbonLocalTransportReceiver" />
    <transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener">
        <parameter name="default" locked="false">
            <parameter locked="false" 
name="java.naming.factory.initial">com.tibco.tibjms.naming.TibjmsInitialContextFactory</parameter>
            <parameter name="java.naming.security.principal" locked="false">userA</parameter>
            <parameter name="java.naming.security.credentials" locked="false">pwduserA</parameter>
            <parameter locked="false" 
name="java.naming.provider.url">tcp://server1:8224,server2:8224</parameter>
            <parameter locked="false" name="transport.jms.UserName">userA</parameter>
            <parameter locked="false" name="transport.jms.Password">pwduserA</parameter>
            <parameter locked="false" 
name="transport.jms.ConnectionFactoryJNDIName">QueueConnectionFactory</parameter>
            <parameter locked="false" name="transport.jms.ConnectionFactoryType">queue</parameter>
        </parameter>
        <parameter locked="false" name="QueueConnectionFactory">
            <parameter locked="false" 
name="java.naming.factory.initial">com.tibco.tibjms.naming.TibjmsInitialContextFactory</parameter>
            <parameter name="java.naming.security.principal" locked="false">userA</parameter>
            <parameter name="java.naming.security.credentials" locked="false">pwduserA</parameter>
            <parameter locked="false" 
name="java.naming.provider.url">tcp://server1:8224,server2:8224</parameter>
            <parameter locked="false" name="transport.jms.UserName">userA</parameter>
            <parameter locked="false" name="transport.jms.Password">pwduserA</parameter>
            <parameter locked="false" 
name="transport.jms.ConnectionFactoryJNDIName">QueueConnectionFactory</parameter>
            <parameter locked="false" name="transport.jms.ConnectionFactoryType">queue</parameter>
        </parameter>
    </transportReceiver>

Everything therefore works perfectly but when (for example) the nominal TIBCO EMS server (server1) switches to the secondary TIBCO EMS server (server2), then my ESB pods report the following error:

javax.naming.ServiceUnavailableException: Failed to query JNDI: Failed to connect to any server at: 
tcp://server1:8224, tcp://server2:8224 [Root exception is javax.jms.JMSException: Failed to connect to any server at: tcp://server1:8224, tcp://server2:8224]
at com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:674)
at com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:494)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at org.apache.axis2.transport.jms.JMSUtils.lookup(JMSUtils.java:667)
at org.apache.axis2.transport.jms.JMSUtils.lookupDestination(JMSUtils.java:895)
at org.apache.axis2.transport.jms.JMSOutTransportInfo.getDestination(JMSOutTransportInfo.java:197)
at 

 org.apache.axis2.transport.jms.JMSOutTransportInfo.loadConnectionFactoryFromProperties
(JMSOutTransportInfo.java:145)
at org.apache.axis2.transport.jms.JMSOutTransportInfo.createJMSSender(JMSOutTransportInfo.java:343)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:135)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:581)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:78)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:548)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:382)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:65)
at org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:55)
at org.apache.synapse.mediators.builtin.CallMediator.handleNonBlockingCall(CallMediator.java:221)
at org.apache.synapse.mediators.builtin.CallMediator.mediate(CallMediator.java:99)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:59)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.rest.Resource.process(Resource.java:356)
at org.apache.synapse.rest.API.process(API.java:399)
at org.apache.synapse.rest.RESTRequestHandler.apiProcess(RESTRequestHandler.java:123)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:101)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:69)
at 
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:304)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:75)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at 

 org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler
(ServerWorker.java:325
at 
org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest
(ServerWorker.java:371)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:151)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.jms.JMSException: Failed to connect to any server at: tcp://server1:8224, 
tcp://server2:8224
at com.tibco.tibjms.TibjmsConnection._create(TibjmsConnection.java:1533)
at com.tibco.tibjms.TibjmsConnection.<init>(TibjmsConnection.java:4412)
at com.tibco.tibjms.TibjmsQueueConnection.<init>(TibjmsQueueConnection.java:39)
at com.tibco.tibjms.TibjmsxCFImpl._createImpl(TibjmsxCFImpl.java:202)
at com.tibco.tibjms.TibjmsxCFImpl._createConnection(TibjmsxCFImpl.java:255)
at com.tibco.tibjms.TibjmsQueueConnectionFactory.createQueueConnection
(TibjmsQueueConnectionFactory.java:90)
at com.tibco.tibjms.naming.TibjmsContext$Messenger.request(TibjmsContext.java:328)
at com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:660)
... 38 more

No more sending of message or reception of JMS message works in this case. You have to restart the ESB pods for it to start again.

I don't understand this problem and would like some help in solving it please. What settings will need to be added or modified or deleted ?

Thanks for your help

-- RBO
docker
kubernetes
wso2
wso2esb

Similar Questions

How to add insecure Docker registry certificate to kubeadm config
Docker container with two jar files , run on demand not as entry point
The PersistentVolume is invalid: spec: Required value: must specify a volume type
kube-proxy daemonset permission problems
Nginx ingress server exposed as nodeport with external ip is reachable from within the cluster, but not from outside

2 Answers

12/22/2020

I see from the above that you are using a connection factory called QueueConnectionFactory.

To be able to manage EMS fail over scenario in a transparent manner you need to configure your factory to manage EMS connection and reconnection parameters.

This can be done with the following tibemsadmin command:

addprop factory QueueConnectionFactory connect_attempt_count=5000
addprop factory QueueConnectionFactory connect_attempt_delay=10000
addprop factory QueueConnectionFactory connect_attempt_timeout=1000
addprop factory QueueConnectionFactory reconnect_attempt_count=5000
addprop factory QueueConnectionFactory reconnect_attempt_delay=10000
addprop factory QueueConnectionFactory reconnect_attempt_timeout=1000
-- EmmanuelM
Source: StackOverflow
12/26/2020

If the reconnection properties are already in place for the connection factories you are using, then the remaining possible explanations would be the following :

. The EMS server FT configuration is not valid, you may need to confirm EMS datastores are shared between the two EMS instances and are managed on a shared volume verifying the four criterias required by EMS (see EMS documentation for this). Also you may need to check the value of the 'ft_reconnect_timeout' parameter (from tibemsd.conf file). By default the value is 60 seconds (one minute) if the client applications are taking time to reconnect this can be useful to increase this value to a few minutes.

. You may also need to check which version and hotfix of the EMS client library you are using with your spring application. I see there are some fixes for EMS failover in EMS 8.3.0 HF04 and 8.4.0 HF02.

-- EmmanuelM
Source: StackOverflow