title: How to ignore an unknown column when loading to bigQuery using Airflow?
body: <p>I'm loading data from Google Storage to bigQuery using <code>GoogleCloudStorageToBigQueryOperator</code></p>
  
  <p>It may be that the Json file will have more columns than what I defined. In that case I want the load job continue - simply ignore this unrecognized column.
  I tried to use the <code>ignore_unknown_values</code> argument but it didn't make any difference.</p>
  
  <p>My operator:</p>
  
  <pre><code>def dc():
      return [
      {
          "name": "id",
          "type": "INTEGER",
          "mode": "NULLABLE"
      },
      {
          "name": "storeId",
          "type": "INTEGER",
          "mode": "NULLABLE"
      },
   ...
  ]
  gcs_to_bigquery_st = GoogleCloudStorageToBigQueryOperator(
      dag=dag,
      task_id='load_to_BigQuery_stage',
      bucket=GCS_BUCKET_ID,
      destination_project_dataset_table=table_name_template_st,
      source_format='NEWLINE_DELIMITED_JSON',
      source_objects=[gcs_export_uri_template],
      ignore_unknown_values = True,
      schema_fields=dc(),
      create_disposition='CREATE_IF_NEEDED',
      write_disposition='WRITE_APPEND',
      skip_leading_rows = 1,
      google_cloud_storage_conn_id=CONNECTION_ID,
      bigquery_conn_id=CONNECTION_ID
  )
  </code></pre>
  
  <p>The error:</p>
  
  <blockquote>
    <p>u'Error while reading data, error message: JSON parsing error in row
    starting at position 0: No such field: shippingService.',</p>
  </blockquote>
  
  <p>which is true. shippingService doesn't exist and it won't be added to the table.</p>
  
  <p>How can I fix this?</p>
  
  <p>Edit:
  Removed the <code>schema_fields=dc()</code> from the operator:</p>
  
  <pre><code>gcs_to_bigquery_st = GoogleCloudStorageToBigQueryOperator(
      dag=dag,
      task_id='load_to_BigQuery_stage',
      bucket=GCS_BUCKET_ID,
      destination_project_dataset_table=table_name_template_st,
      source_format='NEWLINE_DELIMITED_JSON',
      source_objects=[gcs_export_uri_template],
      ignore_unknown_values = True,
      create_disposition='CREATE_IF_NEEDED',
      write_disposition='WRITE_APPEND',
      skip_leading_rows = 1,
      google_cloud_storage_conn_id=CONNECTION_ID,
      bigquery_conn_id=CONNECTION_ID
  )
  </code></pre>
  
  <p>Still gives the same error.
  This doesn't make scene.. It has command to ignore unknown values :( </p>