I want to audit commands that are being run by a user inside a running pod.
I know that kube-apiserver supports audit policies that allows you to log every request that is being done towards the API but as far as I know the API audit only records the exec command and not the inner commands run afterwards.
An approach that I thought is to have a sidecar container with auditbeat running but it's too heavy and the user might be able to kill it.
There are some commercial products that allow to do this. Few weeks ago I did a PoC for one of them. The way it's implemented is that their product running as a pod (with 1 container inside) on the host level (host namespace / HostPID) and tracks usage of Docker daemon.
The container should run a single process inside. It is not recommended to run a command inside container exception for testing. Most of our image doesn't have any type of shell.
If you have to spawn a shell and run a command inside, Then you need to think about is it possible to run that outside container? If the main process is terminated but your shell commands are running in a container then k8s might not terminate that pod and recreate a new pod which might impact HA