I'm implementing a custom resource controller using the watch API. It needs to create/delete objects in aws when objects are created/deleted in kubernetes.
When it starts up the watch, it receives a list of historic events. However, I noticed that if an object is created and then deleted, these events "cancel out" in the historic event stream. That is, when I start a watch, instead of seeing an ADDED event and a DELETED event for the given object, I see no events at all, as though it never existed. This means that if an object is deleted while the controller is down, it will completely miss this delete event when it starts back up.
For controllers that need to take an action when a kubernetes object is deleted (for example, deleting an object in AWS), what is the recommended approach? Is there a way to make kubernetes keep DELETED events? Is it just expected that controllers work by polling the list of all objects in all namespaces rather than using the watch API?
In case you need to synchronize states or list of existing objects in one system with objects in another system, you should be able to get the lists of objects on both systems, compare them, and deal with the difference.
If you rely only on watching instant events like CREATE and DELETE, you will end up with unsynchronized systems sooner or later.
The only reliable source of information about Kubernetes apiserver events I can imagine is the Audit log.