How to configure Non Persistent messages in GCP Pub Sub?

9/12/2021

I have an architecture with multiple pods subscribing to a GCP Topic.

Every pod handles messages while it's up but is not interested in receiving messages it missed when it was not up.

In ActiveMQ this was Non Persistent messages, but I don't see the equivalent in GCP. The only thing I thought is message lifetime with a minimum of 10 minutes.

Is this possible in GCP and where can it be configured ?

-- pmpm
google-cloud-pubsub
java
jms-topic
kubernetes
spring

1 Answer

9/13/2021

There is no option in Cloud Pub/Sub to disable storage. You have two options.

  1. As you suggest, set the message retention duration to the minimum, 10 minutes. This does mean you'll get messages that are up to ten minutes old when the pod starts up. The disadvantage to this approach is that if there is an issue with your pod and it falls more than ten minutes behind in processing messages, then it will not get those messages, even when it is up.

  2. Use the seek operation and seek to seek forward to the current timestamp. When a pod starts up, the first thing it could do is issue a Seek command that would acknowledge all messages before the provided timestamp. Note that this operation is eventually consistent and so it is possible you may still get some older messages when you initially start your subscriber (or, if you are using push, once your endpoint is up). Also keep in mind that the seek operation is an administrative operation and therefore is limited to 6,000 operations a minute (100 operations per second). Therefore, if you have a lot of pods that are often restarting, you may run into this limit.

-- Kamal Aboul-Hosn
Source: StackOverflow