title: Cannot install additional requirements to apache airflow
body: <p>I am using the following docker-compose image, I got this image from: <a href="https://github.com/apache/airflow/blob/main/docs/apache-airflow/start/docker-compose.yaml" rel="noreferrer">https://github.com/apache/airflow/blob/main/docs/apache-airflow/start/docker-compose.yaml</a></p>
  <pre class="lang-yaml prettyprint-override"><code>version: &quot;3&quot;
  x-airflow-common: &amp;airflow-common
    image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.0-python3.7}
    environment: &amp;airflow-common-env
      AIRFLOW__CORE__EXECUTOR: CeleryExecutor
      AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
      AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
      AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
      AIRFLOW__CORE__FERNET_KEY: &quot;&quot;
      AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: &quot;true&quot;
      AIRFLOW__CORE__LOAD_EXAMPLES: &quot;false&quot;
      AIRFLOW__API__AUTH_BACKEND: &quot;airflow.api.auth.backend.basic_auth&quot;
      _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-apache-airflow-providers-apache-spark}
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./plugins:/opt/airflow/plugins
    user: &quot;${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}&quot;
    depends_on: &amp;airflow-common-depends-on
      redis:
        condition: service_healthy
      postgres:
        condition: service_healthy
  
  services:
    postgres:
      image: postgres:13
      environment:
        POSTGRES_USER: airflow
        POSTGRES_PASSWORD: airflow
        POSTGRES_DB: airflow
      volumes:
        - postgres-db-volume:/var/lib/postgresql/data
      healthcheck:
        test: [&quot;CMD&quot;, &quot;pg_isready&quot;, &quot;-U&quot;, &quot;airflow&quot;]
        interval: 5s
        retries: 5
      restart: always
  
    redis:
      image: redis:latest
      ports:
        - 6379:6379
      healthcheck:
        test: [&quot;CMD&quot;, &quot;redis-cli&quot;, &quot;ping&quot;]
        interval: 5s
        timeout: 30s
        retries: 50
      restart: always
  
    airflow-webserver:
      &lt;&lt;: *airflow-common
      command: webserver
      ports:
        - 8080:8080
      healthcheck:
        test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;--fail&quot;, &quot;http://localhost:8080/health&quot;]
        interval: 10s
        timeout: 10s
        retries: 5
      restart: always
      depends_on:
        &lt;&lt;: *airflow-common-depends-on
        airflow-init:
          condition: service_completed_successfully
  
    airflow-scheduler:
      &lt;&lt;: *airflow-common
      command: scheduler
      healthcheck:
        test:
          [
            &quot;CMD-SHELL&quot;,
            'airflow jobs check --job-type SchedulerJob --hostname &quot;$${HOSTNAME}&quot;',
          ]
        interval: 10s
        timeout: 10s
        retries: 5
      restart: always
      depends_on:
        &lt;&lt;: *airflow-common-depends-on
        airflow-init:
          condition: service_completed_successfully
  
    airflow-worker:
      &lt;&lt;: *airflow-common
      command: celery worker
      healthcheck:
        test:
          - &quot;CMD-SHELL&quot;
          - 'celery --app airflow.executors.celery_executor.app inspect ping -d &quot;celery@$${HOSTNAME}&quot;'
        interval: 10s
        timeout: 10s
        retries: 5
      restart: always
      depends_on:
        &lt;&lt;: *airflow-common-depends-on
        airflow-init:
          condition: service_completed_successfully
  
    airflow-init:
      &lt;&lt;: *airflow-common
      command: version
      environment:
        &lt;&lt;: *airflow-common-env
        _AIRFLOW_DB_UPGRADE: &quot;true&quot;
        _AIRFLOW_WWW_USER_CREATE: &quot;true&quot;
        _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
        _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
  
    flower:
      &lt;&lt;: *airflow-common
      command: celery flower
      ports:
        - 5555:5555
      healthcheck:
        test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;--fail&quot;, &quot;http://localhost:5555/&quot;]
        interval: 10s
        timeout: 10s
        retries: 5
      restart: always
      depends_on:
        &lt;&lt;: *airflow-common-depends-on
        airflow-init:
          condition: service_completed_successfully
  
    ######################################################
    # SPARK SERVICES
    ######################################################
  
    jupyterlab:
      image: andreper/jupyterlab:3.0.0-spark-3.0.0
      container_name: jupyterlab
      ports:
        - 8888:8888
        - 4040:4040
      volumes:
        - shared-workspace:/opt/workspace
    spark-master:
      image: andreper/spark-master:3.0.0
      container_name: spark-master
      ports:
        - 8081:8080
        - 7077:7077
      volumes:
        - shared-workspace:/opt/workspace
    spark-worker-1:
      image: andreper/spark-worker:3.0.0
      container_name: spark-worker-1
      environment:
        - SPARK_WORKER_CORES=1
        - SPARK_WORKER_MEMORY=512m
      ports:
        - 8082:8081
      volumes:
        - shared-workspace:/opt/workspace
      depends_on:
        - spark-master
    spark-worker-2:
      image: andreper/spark-worker:3.0.0
      container_name: spark-worker-2
      environment:
        - SPARK_WORKER_CORES=1
        - SPARK_WORKER_MEMORY=512m
      ports:
        - 8083:8081
      volumes:
        - shared-workspace:/opt/workspace
      depends_on:
        - spark-master
  
  volumes:
    postgres-db-volume:
    shared-workspace:
      name: &quot;jordi_airflow&quot;
      driver: local
      driver_opts:
        type: &quot;none&quot;
        o: &quot;bind&quot;
        device: &quot;/Users/jordicrespoguzman/Projects/custom_airflow_spark/spark_folder&quot;
  </code></pre>
  <p>I am trying to run the following DAG:</p>
  <pre class="lang-py prettyprint-override"><code>from airflow import DAG
  from airflow.providers.http.sensors.http import HttpSensor
  from airflow.sensors.filesystem import FileSensor
  from airflow.operators.python import PythonOperator
  from airflow.operators.bash import BashOperator
  
  from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
  from airflow.operators.email import EmailOperator
  
  from datetime import datetime, timedelta
  import csv
  import requests
  import json
  
  default_args = {
      &quot;owner&quot;: &quot;airflow&quot;,
      &quot;email_on_failure&quot;: False,
      &quot;email_on_retry&quot;: False,
      &quot;email&quot;: &quot;admin@localhost.com&quot;,
      &quot;retries&quot;: 1,
      &quot;retry_delay&quot;: timedelta(minutes=5),
  }
  
  
  def printar():
      print(&quot;success!&quot;)
  
  
  with DAG(
      &quot;forex_data_pipeline&quot;,
      start_date=datetime(2021, 1, 1),
      schedule_interval=&quot;@daily&quot;,
      default_args=default_args,
      catchup=False,
  ) as dag:
  
      downloading_rates = PythonOperator(task_id=&quot;test1&quot;, python_callable=printar)
  
      forex_processing = SparkSubmitOperator(
          task_id=&quot;spark1&quot;,
          application=&quot;/opt/airflow/dags/test.py&quot;,
          conn_id=&quot;spark_conn&quot;,
          verbose=False,
      )
  
      downloading_rates  &gt;&gt; forex_processing
  </code></pre>
  <p>But I see this error in the airflow ui:</p>
  <pre class="lang-py prettyprint-override"><code>Broken DAG: [/opt/airflow/dags/dag_spark.py] Traceback (most recent call last):
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 219, in _call_with_frames_removed
    File &quot;/opt/airflow/dags/dag_spark.py&quot;, line 7, in &lt;module&gt;
      from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
  ModuleNotFoundError: No module named 'airflow.providers.apache'
  </code></pre>
  <p>I have specified to install additional requirements in the docker-compose file:</p>
  <pre><code>_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-apache-airflow-providers-apache-spark}
  </code></pre>
  <p>I am writing it wrong? how I should specify the additional requirements I want to install in airflow? can I pass a requirements.txt? if so, how I specify the path?</p>