How to use ytt to add a sidecar container to an existing Kubernetes Deployment?

8/14/2020

I would like to add a fluent-bit agent as a sidecar container to an existing Istio Ingress Gateway Deployment that is generated via external tooling (istioctl). I figured using ytt and its overlays would be a good way to accomplish this since it should let me append an additional container to the Deployment and a few extra volumes while leaving the rest of the generated YAML intact.

Here's a placeholder Deployment that approximates an istio-ingressgateay to help visualize the structure:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  template:
    metadata:
      labels:
        app: istio-ingressgateway
    spec:
      containers:
      - args:
        - example-args
        command: ["example-command"]
        image: gcr.io/istio/proxyv2
        imagePullPolicy: Always
        name: istio-proxy
      volumes:
        - name: example-volume-secret
          secret:
            secretName: example-secret
        - name: example-volume-configmap
          configMap:
            name: example-configmap

I want to add a container to this that looks like:

- name: fluent-bit
  image: fluent/fluent-bit
  resources:
    limits:
      memory: 100Mi
    requests:
      cpu: 10m
      memory: 10Mi
  volumeMounts:
  - name: fluent-bit-config
    mountPath: /fluent-bit/etc
  - name: varlog
    mountPath: /var/log
  - name: dockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true

and volumes that look like:

- name: fluent-bit-config
  configMap:
    name: ingressgateway-fluent-bit-forwarder-config
- name: varlog
  hostPath:
    path: /var/log
- name: dockercontainers
  hostPath:
    path: /var/lib/docker/containers

I managed to hack something together by modifying the overylay files example in the ytt playground, this looks like this:

#@ load("@ytt:overlay", "overlay")
 #@overlay/match by=overlay.subset({"kind": "Deployment", "metadata":{"name":"istio-ingressgateway"}}),expects=1
 ---
 spec:
   template:
     spec:
       containers:
       #@overlay/append
       - name: fluent-bit
         image: fluent/fluent-bit
         resources:
           limits:
             memory: 100Mi
           requests:
             cpu: 10m
             memory: 10Mi
         volumeMounts:
         - name: fluent-bit-config
           mountPath: /fluent-bit/etc
         - name: varlog
           mountPath: /var/log
         - name: dockercontainers
           mountPath: /var/lib/docker/containers
           readOnly: true
 #@overlay/match by=overlay.subset({"kind": "Deployment", "metadata":{"name":"istio-ingressgateway"}}),expects=1
 ---
 spec:
   template:
     spec:
       volumes:
       #@overlay/append
       - name: fluent-bit-config
         configMap:
           name: ingressgateway-fluent-bit-forwarder-config
       #@overlay/append
       - name: varlog
         hostPath:
           path: /var/log
       #@overlay/append
       - name: dockercontainers
         hostPath:
           path: /var/lib/docker/containers

What I am wondering, though, is what is the best, most idiomatic way of using ytt to do this?

Thanks!

-- tcdowney
kubernetes
yaml
ytt

1 Answer

8/14/2020

What you have now is good! The one suggestion I would make is that, if the volumes and containers always need to be added together, they be combined in to the same overlay, like so:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind": "Deployment", "metadata":{"name":"istio-ingressgateway"}}),expects=1
---
spec:
  template:
    spec:
      containers:
      #@overlay/append
      - name: fluent-bit
        image: fluent/fluent-bit
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 10Mi
        volumeMounts:
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc
        - name: varlog
          mountPath: /var/log
        - name: dockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      #@overlay/append
      - name: fluent-bit-config
        configMap:
          name: ingressgateway-fluent-bit-forwarder-config
      #@overlay/append
      - name: varlog
        hostPath:
          path: /var/log
      #@overlay/append
      - name: dockercontainers
        hostPath:
          path: /var/lib/docker/containers

This will guarantee any time the container is added, the appropriate volumes will be included as well.

-- ewrenn
Source: StackOverflow