I'm not a container expert, but my understanding is that the best way to do this is to create a new container, based on your current docker image, that copies the files into place. Your image, with your plugins and configs, is what k8s loads and manages.

FROM my/oldimage:1.7.1
COPY more-plugins/* /opt/myapp/plugins/

$ docker build -t my/newimage:1.7.1 .

So what are the different solutions to this problem? And what is the best one?

The best solution has been already provided by @PaulProgrammer and as long as there are no contraindications you should make your app plugins an integral part of your docker image. The reasoning in favour of such solutions was already well explained by @Matt:

I would only do the copy in a container image build though. It means you always have a complete, reproducible artefact of what is running. – Matt

I fully agree with it and I would also strongly recommend you rebuilding your docker image rather that solve this issue on kubernetes level.

However to make this thread complete and provide you with answers to your additional questions and clear all possible doubts I'd like to add a few words to what was already written.

I'd like to emphasize again that this is not an optimal solution but from technical perspective it is also possible to do it with the help of init containers in kubernetes. Furthermore I will even try to explain why it is not the best solution or more specifically why it is not applicable in your particular usecase.

How do I do that? Is initContainers helpful in such cases? Does init-container have access to the filesystem of the app-container, so that the above command can be executed? I tried to use busybox in initContainers and ran ls -lR /opt/myapp just to see if such a path even exists. But it seems, init container doesn't have access to the app-filesystem.

Keep in mind that init containers are always run before the app containers are started and they cannot modify the original filesystem of the app container as when they run it doesn't even exist yet. However they can be used to pre-populate the additional volume that subsequently can be mounted in the app container. Look at the example below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: debian
  template:
    metadata:
      labels:
        app: debian
    spec:
      containers:
      - name: debian
        image: debian
        command: ['sh', '-c', 'sleep 3600']
        volumeMounts:
         - mountPath: "/app/data"
           name: my-volume
           readOnly: true
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: example-pvc
      initContainers:
      - name: init-myservice
        image: busybox
        command: ['sh', '-c', 'echo "Content of my file" > /mnt/my_file']
        volumeMounts:
         - mountPath: "/mnt"
           name: my-volume

In this example we populate the /app/data directory of the app container with some content with the help of the init container. The command isn't really important here. It can be anything from getting some content using wget to cloning git repo or scp some content from the remote server. Note that you cannot add this way an additional content to the already existing one. If the directory have already existed and had already some content in it, it is wiped out and populated with the new one. Actually it isn't from technical point of view. The directory is used as a new mounpoint for another volume. As it points now to another disk and its content, the original directory content isn't available any more from our perspective.

So what is the typical usecase ? What can it be useful for ?

If your aplication needs some data to process, you typically don't want to make it an integral part of your docker image. Imagine that it changes quite often and you always need to download its most recent version. This way you don't have to worry about rebuilding your images each time.

But this is not applicable in your case. First, because you need to add some data (additional plugins) to something that is already an integral part of your app container image. In this situation init containers won't be very helpful. Second, you rather want to have consistent, reproducible image, containing all the required plugins.

As I said, technically it is feasible as you can upload somewhere all the plugins you need (including the original ones that are already present in the base image) and copy the whole content using an init container. The advantage of such approach is that it doesn't require from you building custom images when the new version of your base app is released. You only change the image tag and init container will take care of populating the /opt/myapp/plugins/ directory with all the required plugins you want and ensure it doesn't contain anything else apart from them.

I hope that it brought something new to this thread and clarified a bit in what situations it is worth using an init container to populate your app container with data and in what situations it is better to stick to your custom docker image.