I have two GKE projects (test and prod) where we deploy the same containers using Helm charts. Both projects use Stackdriver for Logging & Monitoring.
On our test project, I am able to see Stackdriver Monitoring charts across multiple deployments. However, on the production account, I noticed that every time I re-deploy my code, the monitoring overview tab is reset.
See https://nimb.ws/fuhTRt - As you can see the monitor starts at roughly 6 AM, which corresponds to the most recent ReplicaSet deployment at Jun 19, 2019, 5:53:44 AM
This happens even though the deployment workload does not change, only the ReplicaSet and underlying pods change.
How can I persist the monitoring metrics across deployments? - I have very little to go on!