When watching kubernetes resources for changes, what exactly is happening under the hood? Does the http suddenly change to a wss connection?
To solve a problem of too many requests to the kube-apiserver I am rewriting some code to what I think is more of an operator pattern.
In our multi-tenant microservice architecture all services use the same library to look up connection details to tenant-specific DBs. The connection details are saved in secrets within the same namespace as the application. Every tenant DB has its own secret.
So on every call all secrets with the correct label are read and parsed for the necessary DB connection details. We have around 400 services/pods...
My idea: instead of reading all secrets on every call, create a cache and update the cache everytime a relevant secret was changed via a watcher.
My concerns: am I just replacing the http requests with equally expensive websockets? As I understand I will now have an open websocket connection for every service/pod, which still is 400 open connections.
Would it be better to have a proxy service to watch the secrets (kube-apiserver requests) and then all services query that service for connection details (intranet requests, kube-apiserver unreleated)?
From the sources:
// ServeHTTP serves a series of encoded events via HTTP with Transfer-Encoding: chunked
// or over a websocket connection.It pretty much depends on the client which protocol is used (either chunked http or ws), both of them having their cost, which you'll have to compare to your current request frequency.
You may be better of with a proxy cache that either watches or polls in regular intervals, but that depends a lot on your application.