I have a CronJob which runs every 10 minutes and executes some command.
It's failing for some reason and I want to exec
into it to investigate and fix it (There's a 1 time command I need to run to fix a shared volume*).
The issue is when I try to run exec
I get this error, which is expected:
error: cannot exec into a container in a completed pod; current phase is Failed
I would like to create a new pod from the job definition and run a custom command on that (e.g. tail -f
) so that it runs without crashing and I can exec
into it to investigate and fix the issue.
I've been struggling to do this and have only found 2 solutions which both seem a bit hacky (I've used both and they do work, but since I'm still developing the feature I've had to reset a few times)
tail -f
then update the Helm repo and exec
on the new container. Fix the issue and revert back.Pod
YAML file in a directory outside of the Helm repo, with tail -f
. Create it with the kubectl apply -f
command. Then I can exec
on it, do what I need and delete the pod.The issue with the first is that I change the Helm repo. The second requires some duplication and adaptation of code, but it's not too bad.
What I would like is a kubectl
command I can run to do this. Kind of like how you can create a job from a CronJob:
kubectl create job --from=cronjob/myjob myjob-manual
If I could do this to create a pod, or to create a job with a command which never finishes (like tail -f
) it would solve my problem.
*The command I need to run is to pass in some TOTP credentials as a 1 time task to login to a service. The cookies to stay logged in will then exist on the shared volume so I won't have to do this again. I don't want to pass in the TOTP master key as a secret and add logic to interpret it either. So the most simple solution is to set up this service and once in a while I exec
into the pod and login using the TOTP value again.
One more note. This is for a personal project and a tool I use for my own use. It's not a critical service I am offering to someone else so I don't mind if something goes wrong once in a while and I need to intervene.
Looked into this question more, your option 2 is the most viable solution.
Adding a sidecar container - it's the same as option 1, but even more difficult/time consuming.
As mentioned in comments, there are no options for direct imperative pod creation from job
/cronjob
. Available options can be checked for kubectl
:
Also tried out of the interest (logic is to run the command from cronjob
and then continue with specified command), but did not work out:
$ kubectl create job --from=cronjob/test-cronjob manual-job -- tail -f
error: cannot specify --from and command