How to solve SCDF creating Multiple PODS when scheduling a task with SCDF+Kubernetes?

5/29/2020

Hi I run the SCDF in openshift. I have two questions here.

Question 1

When I execute the task using Schedule task option once per minute from SCDF, it creates a pod for each execution in openshift. And the pods are keep running even after the job completed. Assuming there would be jobs running every 15 minutes, I guess lots of pods would be in running mode at the end of a day. I assume there should be a way to move those PODS to completed status once the job returns completed status. But I couldn't figure out how.

Same task running every 1 minute. And the pod Status is running even after job completion

A simple task that I'm using

@Configuration
@EnableBatchProcessing
@EnableTask
public class Facdata_loaderLoader extends BatchJobLoader{

    @Bean
    public JdbcCursorItemReader<> data_loaderReader()
    {
        return JdbcCursorItemReader; // An instance of it
    }

    @Bean
    public Step step01_data_loader()
    {
        return stepBuilderFactory
                .get("S1-CU-Loaddata_loader")
                .<DomainObjects, data_loader> chunk(100)
                .reader(data_loader_Reader())
                .writer(data_loader_writer)
                .listener(listener)
                .allowStartIfComplete(false)
                .startLimit(2)
                .build();
    }

    @Bean
    public Job loaddata_loader()
    {
        return jouilderFactory
                .get(data_loader_JOB_NAME)
                .incrementer(new RunIdIncrementer())
                .start(step_00_job_details_session())
                .next(step_01_data_loader())
                .listener(jobExecutionListener)
                .build();
    }

}

Is there any configuration I'm missing here?

Question 2

Upon running the task once per minute after few executions, the next job execution started throwing error as "job Instance already exists". Exception below.

Caused by: org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={-spring.cloud.task.executionid=63, run.id=5, -spring.cloud.data.flow.platformname=default, -spring.datasource.username=User_name, -spring.cloud.task.name=SingleJob, -spring.datasource.password="Password", -spring.datasource.driverClassName=oracle.jdbc.OracleDriver, -spring.datasource.url="datasource_Url".  If you want to run this job again, change the parameters.
    at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:131)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

In normal spring batch , I could start my job with a jobLauncher and pass the Job_Id as parameter. Like below.

JobParameters param = new JobParametersBuilder()
                            .addString("JobID", String.valueOf(System.currentTimeMillis())).toJobParameters();
jobLauncher.run(facSessionsLoader.loadBBSessions(), param);

But how can I do that in Task when I schedule it through the SCDF ?

-- Rajesh2389
java
kubernetes
spring-batch
spring-boot
spring-cloud-dataflow

0 Answers