Recently I've developed a new application, a Kubernetes controller that runs in my cluster. Its purpose is to automatically refresh all the secrets in the cluster, and the way it performs this task is by retrieving the list of namespaces in the cluster and, for each of them, making an HTTP request to a different service running in the cluster that exposes and endpoint that refreshes all the secrets in a given namespace. In case you are curious, we want to periodically refresh the secrets because we store the secret values in several backends (Hashicorp Vault and GSC buckets, at the time being) and we refer to those values from the k8s secrets manifests by setting as value a custom path to them (eg. custom-prefix://some/path/to/a/secret/data), and at creation time, we have a mutating webhook that translates those paths by the real values taken from the backend.

Apparently, everything is working fine until we reach a point where the pods of my application are stopped and removed from the cluster, and so are the deployment and service resources. I've been trying to understand what happens, but when looking at container logs, the only thing I can see is that they have been gracefully stopped as if someone/another process would have intentionally stopped them.

The kubernetes cluster is running on Amazon EKS and its version is Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

Do you know if there is any way/any log I can take a look at in order to understand why my pods and the deployment behind them are being removed?