K
Q

Running Kubernetes cronjob on a dedicated pod

4/25/2021

I have an API as a Kubernetes service, and I need to run an endpoint in the API every hour. (heavy statistic calculation, takes around 3-5 minutes)

Currently I am using curl to call the end point directly:

containers:
- name: callout
	image: 'buildpack-deps:curl'
	args:
		- /bin/sh
		- '-ec'
		- 'curl http://api-service/v1/Stats/CalculateStats'

The problem is the task is sent to service, and it ends up on one of the pods. Running the calculation keeps the pod busy, and the other requests coming from regular users via front-end slow down.

How can I create a dedicated pod from the same API image with a higher CPU request (so it can run faster) and run the calculation on it, then remove the pod and repeat the process on the next schedule?

Thanks

-- dvdmn
kubernetes
kubernetes-cronjob

Similar Questions

Replication Controller replica ID in an environment variable?
Prevent split-brain/partitioning for RabbitMQ on OpenShift/Kubernetes
Kubernetes Calico node 'XXXXXXXXXXX' already using IPv4 Address XXXXXXXXX, CrashLoopBackOff
pod failed to schedule. openstack over kubernates installation
Caculating the global index for two ranges in Kubernetes Sprig/helm templates?

1 Answer

4/25/2021

This is what came to my mind:

  1. Create a separate deployment for your cronjob task. This deployment would be basically identical to the production deployment, but with a higher CPU claim and (important!) a separate tag. Set the replica count to 0 by default. This deployment can be updated whenever the production deployment is updated, so both deployments run the same version of the appplication.
  2. Create a separate service that binds to all pods to the newly created tag in 1.. This service should not be expopsed to the public (i.e. the internet).

In the cronjob:

  • Scale up the deployment created in 1. to 1
  • Wait for the pod of the deployment to become ready
  • Execute the curl-command
  • Scale down the deployment created in 1. back to 0

This will ensure that the additional resources are only consumed when the expensive request is executed. A downside of this approach is that the deployment of the pod may fail if no sufficient resources are available, so we should monitor the cluster and keep some resource "ready" for when the deployment is scaled up. This can be done, e.g., through a cluster autoscaler on AWS. The scaling may take some time, so the cronjob should be able to handle this kind of delay.

By the by: I would also suggest to secure/hide the endpoint with the "expensive operation" from public, such that no user is able to tear down the production deployment, either "by accident" or out of malicious intentions.

-- Turing85
Source: StackOverflow