title: complicated join in spark: rdd elements have many key-value pairs
body: <p>I'm new to spark and trying to find a way to integrate information from one rdd into another, but their structures don't lend themselves to a standard join function</p>
  
  <p>I have on rdd of this format:</p>
  
  <pre><code>[{a:a1, b:b1, c:[1,2,3,4], d:d1},
   {a:a2, b:b2, c:[5,6,7,8], d:d2}]
  </code></pre>
  
  <p>and another of this format:</p>
  
  <pre><code>[{1:x1},{2,x2},{3,x3},{4,x4},{5,x5},{6,x6},{7,x7},{8,x8}]
  </code></pre>
  
  <p>I want to match the values in the second rdd to their keys in the first rdd (which are in a list value in the c key).  I know how to manipulate them once they're there, so I'm not too concerned about the final output, but I'd maybe like to see something like this:</p>
  
  <pre><code>[{a:a1, b:b1, c:[1,2,3,4],c0: [x1,x2,x3,x4], d:d1},
   {a:a2, b:b2, c:[5,6,7,8],c0: [x5,x6,x7,x8], d:d2}]
  </code></pre>
  
  <p>or this:</p>
  
  <pre><code>[{a:a1, b:b1, c:[(1,x1),(2,x2),(3,x3),(4,x4)], d:d1},
   {a:a2, b:b2, c:[(5,x5),(6,x6),(7,x7),(8,x8)], d:d2}]
  </code></pre>
  
  <p>or anything else that can match the keys in the second rdd with the values in the first.  I considered making the second rdd into a dictionary, which I know how to work with, but I just think my data is too large for that.</p>
  
  <p>Thank you so much, I really appreciate it.</p>