I'm running k8s and I noticed something strange.

one of my services calls a webhook which is integrated with google chats so that if my service receives a request, it notifies me via google chats along with the details of the request.

Now yesterday, I received two notifications from google chats with the details in the request being exactly the same.

I investigated the issue and aggregated the logs from the pods of that particular service and found that the request was in the logs twice.

I then checked the logs for each pod for that service and lo and behold, two pods had logged processing the request. In fact, that particular endpoint creates a row in the database and there it was, two records created, one immediately after the other.

My question is: Is this a common occurrence with k8s? How could this happen? Has this happened to anyone else before?