I understand that Kubernetes CronJobs create pods that run on the schedule specified inside the CronJob. However, the retention policy seems arbitrary and I don't see a way where I can retain failed/successful pods for a certain period of time.
I am not sure about what you are exactly asking here.
CronJob does not create pods. It creates Jobs (which also manages) and those jobs are creating pods. As per Kubernetes Jobs Documentation If the Jobs are managed directly by a higher level controller, such as CronJobs, the Jobs can be cleaned up by CronJobs based on the specified capacity-based cleanup policy. In short, pods and jobs will not be deleted utill you remove CronJob. You will be able to check logs from Pods/Jobs/CronJob. Just use kubctl describe
As default CronJob keeps history of 3 successfulJobs and only 1 of failedJob. You can change this limitation in CronJob spec by parameters:
spec:
successfulJobsHistoryLimit 10
failedJobsHistoryLimit 00 means that CronJob will not keep any history of failed jobs
10 means that CronJob will keep history of 10 succeeded jobs
You will not be able to retain pod from failed job because, when job fails it will be restarted until it was succeded or reached backoffLimit given in the spec.
Other option you have is to suspend CronJob.
kubctl patch cronjob <name_of_cronjob> -p '{"spec:"{"suspend":true}}'If value of spuspend is true, CronJob will not create any new jobs or pods. You will have access to completed pods and jobs.
If none of the above helpd you, could you please give more information what do you exactly expect?
CronJob spec would be helpful.