title: Merge MultiLine records using pyspark
body: <p>I have a dataset like which as a multiline and multi delitmer. I am using spark 2.3 to read the same.</p>
  <p>i wanted to convert into a single line file with a different delimeter</p>
  <pre><code>1223232|*|1212|*|0|*|0|*||*|ABDP|*|1234|*|asda|##|
  12223212|*|1212|*|0|*|0|*||*|ABD
  [c0re] score 
  12-- 12--P|*|1234|*|asda|##|
  1223232|*|1212|*|0|*|0|*||*|ABDP|*|1234|*|asda|##|
  2334343|*|1212|*|0|*|0|*||*|ABD
  [c0re] score 
  12-- 12--P|*|1223|*|asda|##|
  1223232|*|1212|*|0|*|0|*||*|ABDP|*|1234|*|asda|##|
  </code></pre>
  <p>Expected output</p>
  <pre><code>1223232~1212~0~0~~ABDP~1234~asda
  12223212~1212~0~0~~ABD[c0re] score 12-- 12--P~1234~asda
  1223232~1212~0~0~~ABDP~1234~asda
  2334343~1212~0~0~~ABD[c0re] score 12-- 12--P~1223~asda
  1223232~1212~0~0~~ABDP~1234~asda
  </code></pre>
  <p>This file was initially converted using a UNIX sed commands. However the size of the file grew and is 50GB and the servers is being hung.</p>
  <pre><code>while read myline
  do
  sed -i 's/\r//g' $myline
  sed -i ':a;N;$!ba;s/\n//g' $myline # removes new line
  sed -i 's+|\#\#|+\n+g' $myline #replaces all |##| as new line
  sed -i 's/~/-/g' $myline 
  sed -i 's/\\/  /g' $myline
  sed -i 's+|\*|+~+g' $myline # converts all |*| as ~
  sed -i 's+|\#\#|++g' $myline
  done &lt; filename
  </code></pre>
  <p>This has to be rewritten into spark with multichracters delimters and newline in data its notworking out.</p>
  <p>I have the spark code written as but unsure how to treat the |#| as newline or eod instead of newline and reaplce newline with space may be</p>
  <pre><code>df = sc.textFile(source_filename).map(lambda x: x.split(&quot;|*|&quot;)).toDF(header_column)
  </code></pre>