title: How to submit Spark jobs to EMR cluster from Airflow?
body: <p>How can I establish a connection between EMR master cluster(created by Terraform) and Airflow. I have Airflow setup under AWS EC2 server with same SG,VPC and Subnet.</p>
  
  <p>I need solutions so that Airflow can talk to EMR and execute Spark submit.</p>
  
  <p><a href="https://aws.amazon.com/blogs/big-data/build-a-concurrent-data-orchestration-pipeline-using-amazon-emr-and-apache-livy/" rel="noreferrer">https://aws.amazon.com/blogs/big-data/build-a-concurrent-data-orchestration-pipeline-using-amazon-emr-and-apache-livy/</a></p>
  
  <p>These blogs have understanding on execution after connection has been established.(Didn't help much)</p>
  
  <p>In airflow I have made a connection using UI for AWS and EMR:-</p>
  
  <p><a href="https://i.stack.imgur.com/dWfnL.png" rel="noreferrer"><img src="https://i.stack.imgur.com/dWfnL.png" alt="enter image description here"></a></p>
  
  <p>Below is the code which will list the EMR cluster's which are Active and Terminated, I can also fine tune to get Active Clusters:-</p>
  
  <pre class="lang-python prettyprint-override"><code>from airflow.contrib.hooks.aws_hook import AwsHook
  import boto3
  hook = AwsHook(aws_conn_id=‘aws_default’)
      client = hook.get_client_type(‘emr’, ‘eu-central-1’)
      for x in a:
          print(x[‘Status’][‘State’],x[‘Name’])
  </code></pre>
  
  <p>My question is - How can I update my above code can do Spark-submit actions</p>