title: How to fill out nulls according to another dataframe pyspark
body: <p>I currently started using pyspark. I have a two columns dataframe with one column containing some nulls, e.g. </p>
  
  <pre><code>df1
  A             B
  1a3b          7
  0d4s          12
  6w2r          null
  6w2r          null
  1p4e          null
  </code></pre>
  
  <p>and another dataframe has the correct mapping, i.e. </p>
  
  <pre><code>df2
  A             B
  1a3b          7
  0d4s          12
  6w2r          0
  1p4e          3
  </code></pre>
  
  <p>so I want to fill out the nulls in <code>df1</code> using <code>df2</code> s.t. the result is:</p>
  
  <pre><code>A             B
  1a3b          7
  0d4s          12
  6w2r          0
  6w2r          0
  1p4e          3
  </code></pre>
  
  <p>in pandas, I would first create a lookup dictionary from <code>df2</code> then use apply on the <code>df1</code> to populate the nulls. But I'm not really sure what functions to use in pyspark, most of replacing nulls I saw is based on simple conditions, for example, filling all the nulls to be a single constant value for certain column.</p>
  
  <p>What I have tried is:</p>
  
  <pre><code>from pyspark.sql.functions import when, col
  
  df1.withColumn('B', when(df.B.isNull(), df2.where(df2.B== df1.B).select('A')))
  </code></pre>
  
  <p>although I was getting <code>AttributeError: 'DataFrame' object has no attribute '_get_object_id'</code>. The logic is to first filter out the nulls then replace it with the column B's value from df2, but I think <code>df.B.isNull()</code> evaluates the whole column instead of single value, which is probably not the right way to do it, any suggestions?</p>