Logging k8s kubectl commands related activities by user profiles in Splunk

6/15/2020

Disclaimer: I am neither K8s expert and not K8s Administrator and I have limited knowledge in Splunk logs how to access data using Splunk query. So please ignore if you can't help and DON'T close it without understanding what is the ask and I am happy to clarify. This will help people benefited who is running with same questions ask in future. +++++++++++++++++++++++++++++++++++

We are using K8s on-Prem and there are tons of namespaces and users access pretty much every namespaces. Somebody accidentally can issue kubectl delete command to delete anything , it could be pod / service , roles or cluster. My objective in this thread is , is there anyway we can trace who is running every kubectl operations ?

I found below link which can help auditing k8s: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

If Audit is enabled in k8s how can we trace that who has executed every kubectl command operation ? as I said I am neither k8s admin but want to know if there is clear path and way to trace this in logs from k8s back to Splunk ?

Our K8s Admin said audit has been setup already but kubectl command with user details are NOT flowing from Rancher / Fluentd to Splunk . Do we need any specific configuration to turned it on ? which K8s Admin needs to set . Any help would be appreciated .

thanks N.B: - this is open thread from closed one.

-- pauldx
fluentd
kubectl
kubernetes
rancher
splunk

1 Answer

6/26/2020

You can use log backend. With this config all of your audit logs will be logged in disk where fluentd can collect from. This will create logs in master nodes, so fluentd daemonset should be present on them.

-- Akin Ozer
Source: StackOverflow