K
Q

Spring Cloud Data Flow: Docker URI error when running the sample app 'partitioned-batch-job' in Kubernetes cluster

9/13/2021

I modified the dataflow sample app partitioned-batch-job to deploy it in a kubernetes cluster via the SCDF server that is running in the cluster. I use the dashboard to launch this app as a task. The app launches fine, but I see the following URI error for the docker resource I provided to the DeployerPartitionHandler. I've shown the modified code snippet below the error stack for reference. Appreciate any input on whether I am using the right URI syntax, and if so why the master step is unable to launch workers with the provided docker image reference.

java.lang.IllegalArgumentException: Unable to get URI for class path resource [docker:vrajkuma/partitioned-batch-job:2.3.1-SNAPSHOT]
at org.springframework.cloud.deployer.spi.kubernetes.DefaultContainerFactory.create(DefaultContainerFactory.java:80) ~[spring-cloud-deployer-kubernetes-2.6.2.jar:2.6.2]
at org.springframework.cloud.deployer.spi.kubernetes.AbstractKubernetesDeployer.createPodSpec(AbstractKubernetesDeployer.java:210) ~[spring-cloud-deployer-kubernetes-2.6.2.jar:2.6.2]
at org.springframework.cloud.deployer.spi.kubernetes.KubernetesTaskLauncher.launch(KubernetesTaskLauncher.java:237) ~[spring-cloud-deployer-kubernetes-2.6.2.jar:2.6.2]
at org.springframework.cloud.deployer.spi.kubernetes.KubernetesTaskLauncher.launch(KubernetesTaskLauncher.java:119) ~[spring-cloud-deployer-kubernetes-2.6.2.jar:2.6.2]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorker(DeployerPartitionHandler.java:394) [spring-cloud-task-batch-2.3.1-SNAPSHOT.jar:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313) [spring-cloud-task-batch-2.3.1-SNAPSHOT.jar:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302) [spring-cloud-task-batch-2.3.1-SNAPSHOT.jar:2.3.1-SNAPSHOT]
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:106) [spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) [spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) [spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:413) [spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136) [spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) [spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) [spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.3.9.jar:5.3.9]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) [spring-batch-core-4.3.3.jar:4.3.3]

Here is the modified code for the file JobConfiguration.java. I just replaced the maven resource with the docker image reference. Also modified the pom.xml dependency to use the kubernetes deployer (instead of local).

@Configuration
public class JobConfiguration {

private static final int GRID_SIZE = 4;
// @checkstyle:off
@Autowired
public JobBuilderFactory jobBuilderFactory;
@Autowired
public StepBuilderFactory stepBuilderFactory;
@Autowired
public DataSource dataSource;
@Autowired
public JobRepository jobRepository;
// @checkstyle:on
@Autowired
private ConfigurableApplicationContext context;
@Autowired
private DelegatingResourceLoader resourceLoader;
@Autowired
private Environment environment;

@Bean
public PartitionHandler partitionHandler(TaskLauncher taskLauncher, JobExplorer jobExplorer,
										 TaskRepository taskRepository, DockerResourceLoader dockerResourceLoader) throws Exception {
	Resource resource = this.resourceLoader
		.getResource("docker:vrajkuma/partitioned-batch-job:2.3.1-SNAPSHOT");

	DeployerPartitionHandler partitionHandler =
		new DeployerPartitionHandler(taskLauncher, jobExplorer, resource, "workerStep", taskRepository);

The referenced repository:tag is there in docker hub and it is the same URI I provided when registering the app in the dashboard. The k8s setup is a local bare metal cluster running on VMs. Deployed SCDF server via the provided helm chart. I was able to run the other sample apps (billsetuptask & billrun) without any problems in the cluster.

thank you.

-- vrajkuma
docker
kubernetes
spring-batch
spring-cloud-dataflow
spring-cloud-task

Similar Questions

kubernetes pods is forbidden: User "user1" cannot list pods in the namespace "stage"
Kubernetes backup and restore
Openshift Route is not load balancing from Service pods
Kubernetes: Allow pod egress network traffic
Kubernetes probe without blocking requests to POD on failure

1 Answer

9/14/2021

Looks like the DelegatingResourceLoader (this.resourceLoader) is supposed to return a DockerResourceLoader based on the scheme in the URI string. Not sure it is doing that. For now I've just changed the code to explicitly use a DockerResourceLoader bean and it works.

    @Bean
	public DockerResourceLoader getDockerResourceLoader() {
		return new DockerResourceLoader();
	}
	
	@Bean
	public PartitionHandler partitionHandler(TaskLauncher taskLauncher, JobExplorer jobExplorer,
											 TaskRepository taskRepository, DockerResourceLoader dockerResourceLoader) throws Exception {

		Resource resource = dockerResourceLoader.getResource("docker:vrajkuma/partitioned-batch-job:2.3.1-SNAPSHOT");

/*
		Resource resource = this.resourceLoader
			.getResource("docker:vrajkuma/partitioned-batch-job:2.3.1-SNAPSHOT");
 */

		DeployerPartitionHandler partitionHandler =
			new DeployerPartitionHandler(taskLauncher, jobExplorer, resource, "workerStep", taskRepository);
...
-- vrajkuma
Source: StackOverflow