title: Connect airflow to Spark job with Spark submit Operator
body: <p>I would like to run my Spark job in Airflow. From the link <a href="https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/contrib/operators/spark_submit_operator/index.html" rel="nofollow noreferrer">https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/contrib/operators/spark_submit_operator/index.html</a>, I am not sure to understand correctly the sentence:
  &quot;This hook is a wrapper around the spark-submit binary to kick off a spark-submit job. It requires that the “spark-submit” binary is in the PATH or the spark-home is set in the extra on the connection.&quot;
  I have my SPARK_HOME env variable but
  What is spark-submit binary exactly? Is it the .py file with the Spark job?
  The other solution seems to be adding something in the extra of the connection in the Airflow webserver UI. What should be added in that case?</p>
  <p>Any help would be appreciated.</p>
  <p>Edit:
  The error I got is:</p>
  <pre><code>[2020-12-07 01:12:58,817] {base_hook.py:89} INFO - Using connection to: id: spark_default. Host: localhost, Port: None, Schema: None, Login: None, Password: None, extra: XXXXXXXX
  [2020-12-07 01:12:58,821] {spark_submit_hook.py:325} INFO - Spark-Submit cmd: spark-submit --master localhost --name airflow-spark --queue root.default /home/ubuntu/market_risk/utils/etl.py
  [2020-12-07 01:12:58,874] {spark_submit_hook.py:479} INFO - Could not find valid SPARK_HOME while searching ['/', '/home/ubuntu/.local/bin']
  [2020-12-07 01:12:58,874] {spark_submit_hook.py:479} INFO - 
  [2020-12-07 01:12:58,874] {spark_submit_hook.py:479} INFO - Did you install PySpark via a package manager such as pip or Conda? If so,
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - PySpark was not found in your Python environment. It is possible your
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - Python environment does not properly bind with your package manager.
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - 
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - Please check your default 'python' and if you set PYSPARK_PYTHON and/or
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - PYSPARK_DRIVER_PYTHON environment variables, and see if you can import
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - PySpark, for example, 'python -c 'import pyspark'.
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - 
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - If you cannot import, you can install by using the Python executable directly,
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - for example, 'python -m pip install pyspark [--user]'. Otherwise, you can also
  [2020-12-07 01:12:58,875] {spark_submit_hook.py:479} INFO - explicitly set the Python executable, that has PySpark installed, to
  [2020-12-07 01:12:58,876] {spark_submit_hook.py:479} INFO - PYSPARK_PYTHON or PYSPARK_DRIVER_PYTHON environment variables, for example,
  [2020-12-07 01:12:58,876] {spark_submit_hook.py:479} INFO - 'PYSPARK_PYTHON=python3 pyspark'.
  [2020-12-07 01:12:58,876] {spark_submit_hook.py:479} INFO - 
  [2020-12-07 01:12:58,878] {spark_submit_hook.py:479} INFO - /home/ubuntu/.local/bin/spark-submit: line 27: /bin/spark-class: No such file or directory
  [2020-12-07 01:12:58,909] {taskinstance.py:1150} ERROR - Cannot execute: spark-submit --master localhost --name airflow-spark --queue root.default /home/ubuntu/market_risk/utils/etl.py. Error code is: 127.
  Traceback (most recent call last):
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py&quot;, line 984, in _run_raw_task
      result = task_copy.execute(context=context)
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py&quot;, line 187, in execute
      self._hook.submit(self._application)
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py&quot;, line 405, in submit
      self._mask_cmd(spark_submit_cmd), returncode
  airflow.exceptions.AirflowException: Cannot execute: spark-submit --master localhost --name airflow-spark --queue root.default /home/ubuntu/market_risk/utils/etl.py. Error code is: 127.
  [2020-12-07 01:12:58,912] {taskinstance.py:1194} INFO - Marking task as FAILED. dag_id=market_data, task_id=spark_submit_job, execution_date=20201207T010725, start_date=20201207T011258, end_date=20201207T011258
  Traceback (most recent call last):
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py&quot;, line 984, in _run_raw_task
      result = task_copy.execute(context=context)
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py&quot;, line 187, in execute
      self._hook.submit(self._application)
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py&quot;, line 405, in submit
      self._mask_cmd(spark_submit_cmd), returncode
  airflow.exceptions.AirflowException: Cannot execute: spark-submit --master localhost --name airflow-spark --queue root.default /home/ubuntu/market_risk/utils/etl.py. Error code is: 127.
  </code></pre>
  <p>Edit:</p>
  <p>I had 2 installations of spark, one manual (located at <code>/opt/spark</code>) and the other with pip. I removed the pip installation. So now the spark installed in <code>/home/ubuntu/.local/bin'</code> is not a problem anymore but Airflow still does not locate the other manually installed spark.
  (I am still able to do a <code>spark-sumbit file.py</code> directly in the folder where the project is stores.
  The new error is:</p>
  <pre><code>[ 2020-12-07 09:52:06,591] {base_hook.py:89} INFO - Using connection to: id: spark_local. Host: local[*], Port: None, Schema: None, Login: None, Password: None, extra: XXXXXXXX
  [2020-12-07 09:52:06,593] {spark_submit_hook.py:325} INFO - Spark-Submit cmd: spark-submit --master local[*] --name airflow-spark --queue root.default /home/ubuntu/market_risk/utils/etl.py
  [2020-12-07 09:52:06,599] {taskinstance.py:1150} ERROR - [Errno 2] No such file or directory: 'spark-submit': 'spark-submit'
  Traceback (most recent call last):
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py&quot;, line 984, in _run_raw_task
      result = task_copy.execute(context=context)
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py&quot;, line 187, in execute
      self._hook.submit(self._application)
    File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py&quot;, line 395, in submit
      **kwargs)
    File &quot;/usr/lib/python3.6/subprocess.py&quot;, line 729, in __init__
      restore_signals, start_new_session)
    File &quot;/usr/lib/python3.6/subprocess.py&quot;, line 1364, in _execute_child
      raise child_exception_type(errno_num, err_msg, err_filename)
  FileNotFoundError: [Errno 2] No such file or directory: 'spark-submit': 'spark-submit'
  </code></pre>