I'm working on a project that has several event-driven microservices and also using Kubernetes for load balancing. All the services are both publishers and listeners. When a microservice publishes an event, all the listeners are catching the event (if it listens that specific event) and doing their job. There was no problem with this flow until this:
Let's say I have a microservice responsible for sending e-mails. And this service is duplicated 2 times by the load balancer due to high load. Now we have 3 instances of e-mail service. When a "sendMail" event is published, all the 3 instances are catching the event and sending an e-mail for their own. At the end of the day 3 e-mail are being sent.
My question is, can I configure a cloud bus that allows me to publish events for both scenarios. I want to say to an event "when a single listener catches you, disappear" or "go to every listener waiting out there".
For example;
Microservices: A, B, C
Duplicated by load balancer: A1, A2, A3, B1...
Case 1: I want to publish an event for all the services' instances.
Case 2: I want to publish an event for service A instances.
Case 3: I want to publish an event for a single instance of A (don't care which).
I have tried; Giving destinations to events, but all the instances have same bus name since they are duplicated as same. If I give/know a single instance bus name, I wouldn't use it because that pod might die.
Event publishing;
applicationContext().publishEvent(
new customEvent(
this, // Source
busProperties().getId(), // Origin Service
null // Destination Service (null: all)
)
);
Event listener;
@Component
public class Listener implements ApplicationListener<CustomEvent> {
@Override
public void onApplicationEvent(CustomEvent event) {
Foo();
}
}
I have found and implemented a partial solution, using Spring Cloud Stream along side with Spring Cloud Bus. I'm using Cloud Stream with custom queues and groups for single events (sentMail event for the first example). When I send a message through Cloud Stream and if multiple instances of a microservice are listening the message, they receive it one by one like this:
Sending messages: M1, M2, M3
Listening A microservice instances: A1, A2
With Cloud Bus:
A1 receives M1, M2, M3
A2 receives M1, M2, M3
With Cloud Stream (custom groups defined):
A1 receives M1, M3
A2 receives M2
For implementation: https://www.baeldung.com/spring-cloud-stream
I believe this can be solved by setting your consumerGroupId. Each instance of your application(each microservice of email service) needs to have a unique groupId so that rabbitmq will send request to any one of those available unique groups.
I understand better now, Here is the Image Link i posted earlier. I'm sure you already know about it. It is a common issue and the best way is to use a redis acting as a "blocking" mechanism. Since the message queue is sending requests asynchronously to all the consumers, we will not know who is receiving and processing the request but we are ensure that it isn't processed by all by using blocking. Before doing any operation, check for the block and if it doesn't exist, process the request.