title: How to setup S3 connection using .env variables in Airflow using Docker?
body: <p>Previously, a similar question was asked <a href="https://stackoverflow.com/questions/60252680/how-to-programmatically-set-up-airflow-1-10-logging-with-localstack-s3-endpoint">how-to-programmatically-set-up-airflow-1-10-logging-with-localstack-s3-endpoint</a> but it wasn't solved.</p>
  <p>I have Airflow running in Docker container which is setup using docker-compose, I followed this <a href="https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html" rel="nofollow noreferrer">guide</a>. Now I want to download some data from an S3 bucket but I need to setup the credentials to allow that. Everywhere this only seems to be done using the UI by manually setting the <code>AWS_ACCESS_KEY_ID</code> &amp; <code>AWS_SECRET_ACCESS_KEY</code> which exposes these in the UI, I want to set this up in the code itself by reading in the ENV variables. In boto3 this would be done using:</p>
  <pre><code>import boto3
  session = boto3.Session(
      aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
      aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
  )
  </code></pre>
  <p>So how would I do this in the code for the DAGS?</p>
  <p><code>Code:</code></p>
  <pre><code>import traceback
  import airflow
  from airflow import DAG
  from airflow.exceptions import AirflowFailException
  from airflow.operators.python import PythonOperator
  
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook
  
  def _download_s3_data(templates_dict, **context):
      # contains a list of the values returned
      data = templates_dict.get(&quot;sagemaker_autopilot_data&quot;)
      if any([not paths for paths in data]):
          raise AirflowFailException(&quot;Some of the paths were not passed!&quot;)
      else:
          (
              sagemaker_training,
              sagemaker_testing,
          ) = data
          s3hook = S3Hook()
          # parse the s3 url
          bucket_name, key = s3hook.parse_s3_url(s3url=sagemaker_training)
          try:
              # need aws credentials
              file_name = s3hook.download_file(key=key, bucket_name=bucket_name)
          except:
              traceback.print_exc()
              raise AirflowFailException(&quot;Error downloading s3 file&quot;)
  
  </code></pre>
  <p><code>ENV file:</code></p>
  <pre><code>AWS_ACCESS_KEY_ID=
  AWS_SECRET_ACCESS_KEY=
  </code></pre>
  <h3>Edit:</h3>
  <p><a href="https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html" rel="nofollow noreferrer">Amazon Web Services Connection</a> seems to be the only documentation about it but its kinda confusing and doesn't mention how to do this programatically.</p>