title: Which one is a better scheduler in AWS Data Pipeline and AWS SWF
body: <p>I have a situation where I have to trigger my workflow based on this condition <code>"It has to process all files in s3 and then start again when there are files in s3"</code>. However, I found that Data Pipeline starts every scheduled duration while SWF starts and ends the job which also shut downs my EMR Cluster. Both of them are not suitable in this case. So, for a process which has to start or trigger based on a condition neither is suitable is what I found. Is there any alternative? Or is one of SWF and Data Pipeline could perform my task. </p>