Best way to manage ConfigMaps, PVC and Secrets with multiple Helm charts

12/29/2019

I work on a complet digital solution with multiple back-end, front-end and micro-services, running on a IaaS model. We work on a CaaS based infrastructure, with Kubernetes, and Helm for deployments.

A lot of apps share informations that can be find in ConfigMap and Secrets, and they share the same data or tmp PVC.

Actually, I create a chart in each app, with a values-env.yaml for each environments, but I don't know where to put configmap, pvc and secrets files.

Is it correct to create a project with every charts on it, like "sub-charts", with single configmap.yaml, pvc.yaml and secrets.yaml files at the root ?

What's the best way for you ?

-- benCat
kubernetes-helm

1 Answer

12/31/2019

There's really two approaches that can work: pass the actual values into each chart, or create shared objects for these things (either in Helm, or using kubectl apply, or via your CI/deployment tool, or ...) and pass their names into each chart.

If you have something like a master database login that winds up getting shared, you can create a Secret that holds those credentials (again, manually or via some other tool). Each chart's values.yaml would have a reference to that

databaseSecretName: db-credentials

and then it would set up environment variables based on this

env:
  - name: DB_USERNAME
    valueFrom:
      secretKeyRef:
        name: {{ .Values.databaseSecretName }}
        key: username

The most prominent limitation of this is that the Secret must be in the same namespace as the Pod referencing it.

The other approach is to just pass it into the Helm values directly:

env:
  - name: DB_USERNAME
    value: {{ .Values.databaseUsername | required "databaseUsername is required" }}

When you deploy you can use the helm install -f parameter to provide an additional YAML file of override values, that provide the per-environment credentials.

Both of these approaches can work for any of the Kubernetes object types you describe. In the case of Secrets, anyone who can access the Helm state can trivially retrieve them. In Helm 2 this is anyone with access to the Tiller pod; in Helm 3 the same RBAC controls that protect ordinary Secrets can also protect the Secret objects Helm uses internally.


One standard microservices rule is that each service should have isolated data; no service directly accesses another service's data, and all communication is via APIs. Where you talk about sharing a PVC, or my example above talks about sharing database credentials, that's not usually a best practice.

Helm's dependency mechanism flattens things in a way that charts-of-charts don't really work well. Say you have service A that depends on Redis, and service B that also depends on Redis, and the two services expect to each have their own Redises. If you try to have a top-level chart that depends on both A and B, Helm will only install one Redis and share it. That can be a source of trouble.

-- David Maze
Source: StackOverflow