title: Combine multiple rows as JSON object in Pyspark
body: <p>I am very new to pyspark and want to perform following operation on the Data Frame. For rows having similar id I need to combine the associated columns in a JSON block. As shown in example below the output should be 1 JSON block with columns secId, names and path.</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>id</th>
  <th>secId</th>
  <th>names</th>
  <th>path</th>
  <th>bin</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>12</td>
  <td>[{“area” : “en”, “value” : “name1” }, {“area” : “sp”, “value” : “name2”}]</td>
  <td>[abc, xyz]</td>
  <td>bin1</td>
  </tr>
  <tr>
  <td>1</td>
  <td>13</td>
  <td>[{“area” : “en”, “value” : “name3” }, {“area” : “sp”, “value” : “name4”}]</td>
  <td>[klm, nop]</td>
  <td>bin1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Need output as</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>id</th>
  <th>bin</th>
  <th>json</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>bin1</td>
  <td>[{“secId” : 12,“names” : [{“area” : “en”, “value” : “name1” }, {“area” : “sp”, “value” : “name2”}],“path” : [abc, xyz]},{“secId” : 13,“names” : [{“area” :     “en”, “value” : “name3” }, {“area” : “sp”, “value” : “name4”}],“path” : [klm, mno]}]</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><a href="https://i.stack.imgur.com/q2zse.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/q2zse.png" alt="enter image description here" /></a></p>
  <p>It would be helpful if anyone can provide some guidelines on doing this.</p>
  <p>Thank you</p>