title: Combining TSV files to create a new TSV for Apache Arrow table
body: <p>I have two TSV files (header.tsv &amp; data.tsv) header.tsv holds 1000+ column names and data.tsv holds ~50K records (with NULL column values too). I would like to create a new TSV file (let's say combined.tsv) by appending data.tsv file to header.tsv. The reason for doing this is to create one final TSV file where it can hold both column names and data together and try to see if I can avoid errors while creating an Apache Arrow table.</p>
  <pre><code>**header.tsv**
  field1 field2 field3 field4 ... field1000 
  
  **data.tsv**
  eng-en    1er2p  NULL  ert,yu1  ...  2020-09-16
  frnch-fr  2er3p  NULL  ert,yu2  ...  2020-09-16
  .
  .
  .
  ltn-lt    50Ker  NULL  ert,yu50K ... 2020-09-16
  </code></pre>
  <p>Required TSV</p>
  <pre><code>**combined.tsv**
  field1    field2   field3   field4    ...   field1000
  eng-en    1er2p    NULL     ert,yu1   ...   2020-09-16
  frnch-fr  2er3p    NULL     ert,yu2   ...   2020-09-16
  .
  .
  .
  ltn-lt    50Ker    NULL     ert,yu50K ...   2020-09-16
  </code></pre>
  <p>I've used the SHELL commands like</p>
  <pre><code>paste header.tsv data.tsv &gt; combined.tsv
  </code></pre>
  <p>and then tried to create a pyarrow table.</p>
  <pre><code>import pyarrow as pa
  import pyarrow.csv as csv
  combined = csv.read_csv('combined.tsv',parse_options=csv.ParseOptions(delimiter=&quot;\t&quot;))
  </code></pre>
  <p>I get below error while executing the above</p>
  <pre><code>ArrowInvalid: CSV parse error: Expected 2010 columns, got 1006
  </code></pre>
  <p>The header.tsv file has exactly 1005 columns and this header.tsv file can be parsed to create a pyarrow table but not the data.tsv file.</p>
  <pre><code>import pyarrow as pa
  import pyarrow.csv as csv
  header = csv.read_csv('headers.tsv',parse_options=csv.ParseOptions(delimiter=&quot;\t&quot;)) 
  head_show=header.to_pandas()
  head_show.head()
  </code></pre>
  <p>I've even tried to use a concat_tables method from pyarrow by doing below</p>
  <pre><code>import pyarrow as pa
  final_combined = pa.concat_tables(header,data)
  </code></pre>
  <p>Error</p>
  <pre><code>TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Table
  </code></pre>
  <p>Please correct me if my approach is wrong.</p>