K
Q

Trigger a Java REST endpoint upon S3 File Upload Event

2/26/2020

I have a Spring Batch application which does followings,

  1. A different application puts a csv file in a S3 bucket INBOUND
  2. Spring Batch application reads a csv file
  3. using data of each row, it calls an API endpoint and get a quote
  4. Put those quotes in a newly created csv and put in a S3 bucket OUTBOUND

Spring Batch Application Tech Stack

  • Created using Spring Boot

  • Spring Batch Job Launcher is called upon a REST endpoint

  • Deployed in OpenShift

What I want to do,

When the other application puts a new file into the S3 INBOUND bucket it needs to trigger the Spring Batch Operation (call the REST endpoint which launches the job)

Can I do this using a K8s Cron Job which triggered by S3 events? If so can someone please explain how? Or do I have any other options?

-- samme4life
amazon-s3
kubernetes
kubernetes-cronjob
rest
spring-batch

Similar Questions

GCP HTTP(S) load balancer ignoring GKE readinessProbe specification
Is mTLS necessary for ClusterIP services on GKE?
Exposing service in (GKE) Kubernetes only with internal ip
Kubernetes DaemonSet only on master nodes

2 Answers

3/3/2020

I am not too familiar with AWS services but I know that they provide a notification service for S3 buckets: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-event-notifications.html

One of the event notification destinations is AWS Lambda. So you can try to create a function that calls the rest endpoint and which is triggered when a notification of type ObjectCreated arrives.

-- Mahmoud Ben Hassine
Source: StackOverflow
3/23/2020

My solution was to create a lambda function which is triggered when a .txt file is uploaded into the S3 bucket's INBOUND folder.

Wrote a python handler to curl command to hit the job launcher REST end point.

-- samme4life
Source: StackOverflow