title: How to modify the source file path of a BigQuery external table in Airflow?
body: <p>There is a process which exports some CSV files to GCS and puts the current datetime into the path, e.g.:</p>
  <pre><code>gs://my_gcs_bucket/path/to/export/&lt;current_timestamp&gt;/exported_file_1.csv
  gs://my_gcs_bucket/path/to/export/&lt;current_timestamp&gt;/exported_file_2.csv
  </code></pre>
  <p>I want to create an external table for these files before I cast its columns and merge with other tables.</p>
  <p>The problem is that I can't implement a DAG in Airflow which can handle the changing timestamps.</p>
  <hr />
  <p>I can create an external table by specifying the path (with the current timestamp), but for tomorrow's exports this external table will not read the new files. But I will need tomorrow the same <code>project_name.dataset.tablename</code> for processing.</p>
  <pre><code>from airflow.contrib.operators.bigquery_operator import BigQueryCreateExternalTableOperator
  
  CreateExternalBqTable = BigQueryCreateExternalTableOperator(
      task_id                             = &quot;CreateExternalBqTable&quot;,
      field_delimiter                     = '|',
      destination_project_dataset_table   = f'{project_name}.{dataset}.{tablename}',
      bucket                              = BUCKET_URI,
      source_objects                      = [ 'path/to/export/2021-12-12-12-12-12/exported_file_*' ],
      schema_fields                       = generate_custom_struct()
  )
  </code></pre>
  <p>If tomorrow I'll try to run the same table creation task again it will fail because the external table already exists.
  I can delete the existing external table then recreate it again to ensure it does not fail the external table creation task, but if it's already deleted then this will fail:</p>
  <pre><code>from airflow.providers.google.cloud.operators.bigquery import BigQueryDeleteTableOperator
  
  DeleteExternalBqTable = BigQueryDeleteTableOperator(
      task_id                 = &quot;DeleteExternalBqTable&quot;,
      deletion_dataset_table  = f'{project_name}.{dataset}.{tablename}',
  )
  </code></pre>
  <hr />
  <p>What is the good pattern here?<br />
  Should I <strong>always delete and recreate the external table</strong> just because of the new timestamps?<br />
  It seems to me very bad and error prone pattern.</p>
  <p>Or could I <strong>change the path under an existing external table</strong> somehow?</p>
  <p>I would be much happier if I could <strong>separate the initialization and deletion phases of my external table</strong> from the daily runs and not always delete or create them. I plan to <strong>initialize these tables only once</strong> and move away the CSV files if I'm done with the processing and keep the external tables empty until the next run.</p>
  <p>(I need a working solution for Airflow 1.x)</p>