I have a kubernetes cluster which processes a series of jobs creating result files. I would like to sync these results to an S3 folder upon completion of all jobs. Is there a way to schedule an S3 sync after all jobs have been completed then shut down all of the instances in the cluster?
I can think of 2 ways you can put the result files in s3.
Let your job program (code) handle putting the results to s3 by using amazon SDK. https://aws.amazon.com/tools/ You can give pods access to the s3 as mentioned here https://stackoverflow.com/a/57668823/3514300
Let your job write to local and you create a ReadWriteMany type of PV in Kubernetes using EFS (https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs) and you copy files from EFS to s3 once every job is done.