title: How to fix the exception 'Invalid argument, not a string or column' while joining two dataframes in Pyspark?
body: <p>I am trying to join two dataframes:</p>
  <p>df1, columns:</p>
  <pre><code>document_nbr, line_id, product_id, product_size, reference_nbr, local_cd, invoice_local_cost, invoice_delivery_id, created_by_id, transaction_ind, etl_tmst, record_created_tmst, record_updated_tmst, source_id, line_number, etl_date
  </code></pre>
  <p>Joining df1 with df2:</p>
  <p>df2 columns:</p>
  <pre><code>document_nbr, line_id, variant_id, line_nbr as line_number
  </code></pre>
  <p>on columns:</p>
  <pre><code>df1.document_nbr = df2.document_nbr and df1.line_id == df2.line_id
  </code></pre>
  <p>Join type: <code>left</code></p>
  <p>I have a column <code>line_number</code> present in both the dataframes <code>df1</code> and <code>df1</code> and I want to drop same column from <code>df1</code> and select it from <code>df2</code>.</p>
  <p>To do that, I wrote the below logic to select all the columns but <code>line_number</code> from <code>df1</code> and select <code>line_number</code> from <code>df2</code></p>
  <pre><code>df1 = df1.join(df2.alias('df2'),
      on=(df1.document_nbr == df2.document_nbr) &amp; (df1.line_id == df2.line_id),
      how='left').\
      select([col(c) for c in df1.columns if c != 'line_number'], col(&quot;df2.line_number&quot;))
  </code></pre>
  <p>When I run the code, I get the exception:</p>
  <pre><code>    Invalid argument, not a string or column: [Column&lt;b'document_nbr'&gt;, Column&lt;b'line_id'&gt;, Column&lt;b'product_id'&gt;, Column&lt;b'product_size'&gt;, Column&lt;b'reference_nbr'&gt;, Column&lt;b'local_cd'&gt;, Column&lt;b'invoice_local_cost'&gt;, Column&lt;b'invoice_delivery_id'&gt;, Column&lt;b'created_by_id'&gt;, Column&lt;b'transaction_ind'&gt;, Column&lt;b'etl_tmst'&gt;, Column&lt;b'record_created_tmst'&gt;, Column&lt;b'record_updated_tmst'&gt;, Column&lt;b'source_id'&gt;, Column&lt;b'source_id'&gt;,Column&lt;b'etl_date'&gt;] of type &lt;class 'list'&gt;. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.
  Traceback (most recent call last):
    File &quot;/local_disk0/tmp/spark-14a2e8a1-b280-4c83-a724-16976b1bc277/template_python-1.0.0-py3-none-any.whl/src/line_upd.py&quot;, line 66, in line_transformation
      how='left').select([col(c) for c in df1.columns if c != 'line_number'], col(&quot;df2.line_number&quot;))
    File &quot;/databricks/spark/python/pyspark/sql/dataframe.py&quot;, line 1439, in select
      jdf = self._jdf.select(self._jcols(*cols))
    File &quot;/databricks/spark/python/pyspark/sql/dataframe.py&quot;, line 1233, in _jcols
      return self._jseq(cols, _to_java_column)
    File &quot;/databricks/spark/python/pyspark/sql/dataframe.py&quot;, line 1220, in _jseq
      return _to_seq(self.sql_ctx._sc, cols, converter)
    File &quot;/databricks/spark/python/pyspark/sql/column.py&quot;, line 68, in _to_seq
      cols = [converter(c) for c in cols]
    File &quot;/databricks/spark/python/pyspark/sql/column.py&quot;, line 68, in &lt;listcomp&gt;
      cols = [converter(c) for c in cols]
    File &quot;/databricks/spark/python/pyspark/sql/column.py&quot;, line 56, in _to_java_column
      &quot;function.&quot;.format(col, type(col)))
  </code></pre>
  <p>Could anyone let me know what is the mistake I am doing here ? How can I select all columns but <code>line_number</code> from <code>df1</code> abd column <code>line_number</code> from <code>df2</code> after the join ?</p>