This might not have to do anything with K8s pods, but something more algorithmic I suspect. But this is the complete scenario we are facing.
Lets say we have 2 pods -> running java application.
We have 1 Dynamo table -> having id(hash_key)(not unique), created_date(sort_key), id_2
The expected behavior of the program is to check the existence of given id (latest) and fetch its id_2. If no such id exists then generate a new id_2.
Now comes the race condition --> Two pods parallelly starts to execute the logic and both of them starts to query the Dynamo, and coincidentally have the same id. Now they do not find any such id .. since none of them have been inserted into Dynamo and therefore they create altogether separate new id_2..and both pods ends up inserting new id_2 for same id.. which should not be the case.
How can we solve this race condition.
Any leads will be appreciated. Thx
I think you shoukd re-think your table schema.
By having a table with id_2 as hash, no sort key and a global secondary index on id/created_data
Of couse this would only work if id_2 is unique in the table, not unique for each id