title: AWS data pipeline VS lambda for EMR automation
body: <p>Here are the steps for my application in AWS .</p>
  
  <ol>
  <li>Data will be loaded weekly in separate 35  <code>S3</code> folders .</li>
  <li>On completion of data loading in each 35 folders 35 EMR cluster will be created .</li>
  <li>Each EMR cluster will have spark-scala script to run parrelly .</li>
  <li>On completion of job all cluster will be terminated .</li>
  </ol>
  
  <p>How can i achieve this ?</p>
  
  <p>As far as i have searched there are two options .</p>
  
  <ol>
  <li>Invoking <code>AWS lambda</code> function on S3 event and lambda will create EMR cluster and will do spark-submit .</li>
  <li>I read about AWS data pipeline .</li>
  </ol>
  
  <p>Will <code>AWS Data pipeline</code> will be helpful in my scenario ?</p>
  
  <p>Also i have spark-scala script that i have been running zeppelin .
  If required i can create jar out of that and submit in data pipe line .</p>
  
  <p>Please consider the cost also .I have 5TB of data to be delivered to client weekly .</p>