title: map values in a dataframe from a dictionary using pyspark
body: <p>I want to know how to map values in a specific column in a dataframe. </p>
  
  <p>I have a dataframe which looks like:</p>
  
  <pre><code>df = sc.parallelize([('india','japan'),('usa','uruguay')]).toDF(['col1','col2'])
  
  +-----+-------+
  | col1|   col2|
  +-----+-------+
  |india|  japan|
  |  usa|uruguay|
  +-----+-------+
  </code></pre>
  
  <p>I have a dictionary from where I want to map the values.</p>
  
  <pre><code>dicts = sc.parallelize([('india','ind'), ('usa','us'),('japan','jpn'),('uruguay','urg')])
  </code></pre>
  
  <p>The output I want is:</p>
  
  <pre><code>+-----+-------+--------+--------+
  | col1|   col2|col1_map|col2_map|
  +-----+-------+--------+--------+
  |india|  japan|     ind|     jpn|
  |  usa|uruguay|      us|     urg|
  +-----+-------+--------+--------+
  </code></pre>
  
  <p>I have tried using the <a href="https://spark.apache.org/docs/latest/api/python/pyspark.html?highlight=sort#pyspark.RDD.lookup" rel="noreferrer"><code>lookup function</code></a> but it doesn't work. It throws error SPARK-5063. Following is my approach which failed:</p>
  
  <pre><code>def map_val(x):
      return dicts.lookup(x)[0]
  
  myfun = udf(lambda x: map_val(x), StringType())
  
  df = df.withColumn('col1_map', myfun('col1')) # doesn't work
  df = df.withColumn('col2_map', myfun('col2')) # doesn't work
  </code></pre>