title: read pretty json format data through spark
body: <p>We read data present in hour format present in S3 through spark in scala.For example,</p>
  <blockquote>
  <p>spark.read.textFile(&quot;s3://'Bucket'/'key'/'yyyy'/'MM'/'dd'/'hh'/*&quot;).</p>
  </blockquote>
  <p>spark.read.textFile reads records one line at a time so for example records that are present in jsonLines(full json data in one line) are read and can be parsed later to retrieve data from json.</p>
  <p>Now,I have to read data which is having multiple json but in pretty format instead of json lines.Using same strategy gives corrupt record error.For example Dataset[String] obtained after reading through spark.read.textFile:</p>
  <pre><code>{
  &quot;a&quot;: 1, 
  &quot;b&quot;: 2
    }
  </code></pre>
  <p>is</p>
  <pre><code>_corrupt_record|
    +---------------+
    |              {|
    |       &quot;a&quot;: 1, |
   |         &quot;b&quot;: 2|
     |              }|
  </code></pre>
  <p>Input data :</p>
  <pre><code>{
  &quot;key1&quot;: &quot;value1&quot;,
  &quot;key2&quot;: &quot;value2&quot;
  }
  {
  &quot;key1&quot;: &quot;value1&quot;,
   &quot;key2&quot;: &quot;value2&quot;
  }
  </code></pre>
  <p>ExpectedOutput</p>
  <pre><code>+------+------+
  |key1  |key2  |
  +------+------+
  |value1|value2|
  |value1|value2|
  +------+------+
  </code></pre>
  <p>This file has multiple pretty formatted json with delimiter between records as newline.</p>
  <p><strong>Approaches already used</strong></p>
  <ul>
  <li>spark.read.option(&quot;multiline&quot;, &quot;true&quot;).json(&quot;&quot;) .This will not work as multiline requires data to be present in form of [{},{}].</li>
  </ul>
  <p><strong>Approach working</strong></p>
  <pre><code>val x=sparkSession
  .read
  .json(sc
    .wholeTextFiles(filePath)
    .values
    .flatMap(x=&gt; {x
    .replace(&quot;\n&quot;, &quot;&quot;)
     .replace(&quot;}{&quot;, &quot;}}{{&quot;)
     .split(&quot;\\}\\{&quot;)}))
  </code></pre>
  <p>I just wanted to ask if there is a better approach as the above solution is doing some slice and dice on data which might lead to performance issue for large data?Thanks</p>