I'm running a job with a large number of completions, so hundreds of pods are created and eventually move into COMPLETED status.
Is there a way to expire the pods as they complete so I don't end up with a massively long list of completed pods?
We can do this for jobs using:
spec:
ttlSecondsAfterFinished: 300
successfulJobsHistoryLimit: 1But I haven't seen a way to do that for the pods created by the job.
No, they're left behind on purpose and you have to clean them up manually. We have a custom cleaner job which removes them after 7 days.
When a Job completes, no more Pods are created, but the Pods are not deleted either. Keeping them around allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output. The job object also remains after it is completed so that you can view its status. It is up to the user to delete old jobs after noting their status.
https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup