I am creating some secrets when helm install is executed via pre-install hooks.
Everything works great. However when helm delete is performed the secrets created are not deleted. This is because any resource installed using pre-install is considered to be self managed. So I read this could be done using post-delete hooks.
So questions are:
How do I delete secrets in post delete?
If we remove pre-install hooks then then delete works just fine. But then how to guarantee that secrets are created before the pods are even created when we perform helm install?
You can add this line.
"helm.sh/hook-delete-policy": "hook-succeeded"Tiller creates resources in a specific order (find it in the source code here: https://github.com/kubernetes/helm/blob/master/pkg/tiller/kind_sorter.go#L26)
So for this specific user case there is no need for hooks or any other mechanism, just include your secret and your pods and magic will happen ;)
That said, there is still the issue with pre-installed objects. The documentation states that this is the desired behaviour:
Practically speaking, this means that if you create resources in a hook, you cannot rely upon helm delete to remove the resources. To destroy such resources, you need to either write code to perform this operation in a pre-delete or post-delete hook or add "helm.sh/hook-delete-policy" annotation to the hook template file.
The only solution is to add a job to the chart, with the post-delete hook, that deletes those resources.
You can use any image with kubectl installed to delete this with pre-delete hook, like @ignacio-millán mentioned. For this, you will also need a role to run this job privileged so it can delete secrets.
Or you can just use the K8s REST API and curl it like this:
apiVersion: batch/v1
kind: Job
metadata:
labels:
annotations:
"helm.sh/hook": pre-delete # << run before delete (we still need the role)
...
spec:
template:
spec:
serviceAccountName: your-privileged-serviceaccount
automountServiceAccountToken: true
# this will mount var/run/secrets/kubernetes.io/serviceaccount/token
containers:
- name: pre-delete
image: "appropriate/curl" # alpine + curl (3 MB)
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
command:
- "/bin/sh"
- "-ec"
- |
curl -s -X DELETE -k https://kubernetes.default.svc/api/v1/namespaces/${NAMESPACE}/secrets/your-secret-name-nere \
-H "Authorization: Bearer $( cat /var/run/secrets/kubernetes.io/serviceaccount/token )" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-d "{ \"kind\": \"Secret\", \"apiVersion\": \"v1\", \"metadata\": { \"name\": \"your-secret-name-nere\", \"namespace\": \"${NAMESPACE}\" } }" > /dev/null
{{- end }}We need to "RoleBind" a ServiceAccount with a Role (or clusterRole). Use helm.sh/hook-weight for ordering all this. the role should have something like this:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role # Roles works only in namespace vs ClusterRole works in all the cluster (warning!)
metadata:
namespace: {{ .Release.Namespace }}
annotations:
"helm.sh/hook": pre-install
"helm.sh/hook-weight": "20"
"helm.sh/hook-delete-policy": before-hook-creation
labels:
....
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "watch", "list", "create", "update"]Note we forward the curl output to null with > /dev/null, we don't want this in the logs. ;) comment this and "helm.sh/hook-delete-policy" for debugging.
There is a working in progress here with something like this applied here:
https://github.com/Flag5/consul-helm/tree/tls-encryption/templates see tls-*.yaml files.
hope it helps :)