title: How to check if array column is inside another column array in PySpark dataframe
body: <p>Suppose I have the following case</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql.types import *
  schema = StructType([  # schema
      StructField("id", StringType(), True),
      StructField("ev", ArrayType(StringType()), True),
      StructField("ev2", ArrayType(StringType()), True),])
  df = spark.createDataFrame([{"id": "se1", "ev": ["ev11", "ev12"], "ev2": ["ev11"]},
                              {"id": "se2", "ev": ["ev11"], "ev2": ["ev11", "ev12"]},
                              {"id": "se3", "ev": ["ev21"], "ev2": ["ev11", "ev12"]},
                              {"id": "se4", "ev": ["ev21", "ev22"], "ev2": ["ev21", "ev22"]}],
                             schema=schema)
  </code></pre>
  
  <p>Which gives me:</p>
  
  <pre><code>df.show()
  +---+------------+------------+
  | id|          ev|         ev2|
  +---+------------+------------+
  |se1|[ev11, ev12]|      [ev11]|
  |se2|      [ev11]|[ev11, ev12]|
  |se3|      [ev21]|[ev11, ev12]|
  |se4|[ev21, ev22]|[ev21, ev22]|
  +---+------------+------------+
  </code></pre>
  
  <p>I want to create a new column of boolean (or select only the true cases) for the rows where the contents of the "ev" column are inside the "ev2" column, returning:
  </p>
  
  <pre><code>df_target.show()
  </code></pre>
  
  <pre class="lang-none prettyprint-override"><code>+---+------------+------------+
  | id|          ev|         ev2|
  +---+------------+------------+
  |se2|      [ev11]|[ev11, ev12]|
  |se4|[ev21, ev22]|[ev21, ev22]|
  +---+------------+------------+
  </code></pre>
  
  <p>or:</p>
  
  <pre class="lang-py prettyprint-override"><code>df_target.show()
  </code></pre>
  
  <pre class="lang-none prettyprint-override"><code>+---+------------+------------+-------+
  | id|          ev|         ev2|evInEv2|
  +---+------------+------------+-------+
  |se1|[ev11, ev12]|      [ev11]|  false|
  |se2|      [ev11]|[ev11, ev12]|   true|
  |se3|      [ev21]|[ev11, ev12]|  false|
  |se4|[ev21, ev22]|[ev21, ev22]|   true|
  +---+------------+------------+-------+
  </code></pre>
  
  <p>I tried using the <code>isin</code> method:</p>
  
  <pre class="lang-py prettyprint-override"><code>df.withColumn('evInEv2', df['ev'].isin(df['ev2'])).show()
  </code></pre>
  
  <pre class="lang-none prettyprint-override"><code>+---+------------+------------+-------+
  | id|          ev|         ev2|evInEv2|
  +---+------------+------------+-------+
  |se1|[ev11, ev12]|      [ev11]|  false|
  |se2|      [ev11]|[ev11, ev12]|  false|
  |se3|      [ev21]|[ev11, ev12]|  false|
  |se4|[ev21, ev22]|[ev21, ev22]|   true|
  +---+------------+------------+-------+
  </code></pre>
  
  <p>But it looks like it only checks if it's the same array.</p>
  
  <p>I also tried the <code>array_contains</code> function from <code>pyspark.sql.functions</code> but only accepts one object and not an array to check.</p>
  
  <p>I am having difficulties even searching for this due to phrasing the correct problem.</p>
  
  <p>Thanks!</p>