How to automateTalend Job2Docker jobs to Kubernetes

4/12/2019

Can someone give me some guidance on best practice to bring multiple Talend jobs dynamically into Kubernetes?

  1. I am using Talend Open Studio for Big Data
  2. I have a listener server for Job2Docker
  3. How do I change the scripts to automate a push to Docker Hub?
  4. Is it possible to have a dynamic CronJob K8s type that can run jobs based on a configuration file.
-- Anton Swanevelder
kubernetes
talend

1 Answer

5/29/2019

I ended up not using Job2Docker in favour of a simple docker process.

  1. Build the Talend Standalone job.
  2. Unzip the build into a folder called jobs.
  3. Use the below Dockerfile example to build and push your docker image.

Dockerfile

FROM java
WORKDIR /talend
COPY ./jobs /talend

Create a CronJob type for K8s

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: etl-edw-cronjob
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: etl-edw-job
              image: dockerhubrepo/your-etl
              command: ["sh",  "./process_data_warehouse_0.1/process_data_warehouse/process_data_warehouse_run.sh"]
              env:
                - name: PGHOST
                  value: postgres-cluster-ip-service
                - name: PGPORT
                  value: "5432"
                - name: PGDATABASE
                  value: infohub
                - name: PGUSER
                  value: postgres
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: pgpassword
                      key: PGPASSWORD
                - name: MONGOSERVER
                  value: mongo-service
                - name: MONGOPORT
                  value: "27017"
                - name: MONGODB
                  value: hearth
-- Anton Swanevelder
Source: StackOverflow