We currently run a number of our websites in a single Kubernetes cluster.

The config for the whole cluster and each of the services within it is contained in a set of configuration files that we keep in a "deployment-configs" repository, so that changes to that config can be reviewed using a standard Git workflow, before being applied to the cluster.

So at the moment, we have Jenkins jobs that kubectl apply -fs effectively all the files in that repository, which we run whenever something changes.

This feels rather brittle - if somehow a critical breaking change made its way into that repository then running that Jenkins job could bring down all of our production sites in one fell swoop.

The alternative is to selectively apply bits of the config when they change - e.g. one per service. The downside here is that we increase the chance that the actual shape of the cluster would be more likely to diverge from that described by the repository.

Is there a way to ask Kubernetes exactly what changes are going to be applied before actually applying them? This would help, 'cos we could do a checking step before applying the config to the whole cluster.

Or does anyone know of any other strategies for managing the config for a Kubernetes cluster?