title: Unable to use jsonlite::stream_in with certain JSON formats
body: <p>I'm attempting to stream the rather large (65gb) JSON file from the YASP data dump (<a href="https://github.com/yasp-dota/yasp/wiki/JSON-Data-Dump" rel="nofollow">https://github.com/yasp-dota/yasp/wiki/JSON-Data-Dump</a>), but it seems that the way the JSON file is formatted means that i cannot read the file, and gives this error:</p>
  
  <blockquote>
    <p>Error: parse error: premature EOF
                                          [
                        (right here) ------^</p>
  </blockquote>
  
  <p>I've created this small sample JSON file using the same format so everyone else can recreate it easily:</p>
  
  <pre><code>[
  {"match_id": 2000594819,"match_seq_num": 1764515493}
  ,
  {"match_id": 2000594820,"match_seq_num": 1764515494}
  ,
  {"match_id": 2000594821,"match_seq_num": 1764515495}
  ]
  </code></pre>
  
  <p>I've saved this file as test.json, and attempt to load it through the jsonlite::stream_in function</p>
  
  <pre><code>library(jsonlite)
  con &lt;- file('~/yasp/test.json')
  jsonStream &lt;- stream_in(con)
  </code></pre>
  
  <p>I get the same "premature EOF" error as shown above.</p>
  
  <p>However, if the format of the file is all in one single chunk like so:</p>
  
  <pre><code>[{"match_id": 2000594819,"match_seq_num": 1764515493},{"match_id": 2000594820,"match_seq_num": 1764515494},{"match_id": 2000594821,"match_seq_num": 1764515495}]
  </code></pre>
  
  <p>Then there are no issues, and the stream_in works fine.</p>
  
  <p>I've played around with using readLines, and collapsing the frame before reading it in:</p>
  
  <pre><code>initialJSON &lt;- readLines('~/yasp/test.json')
  collapsedJSON &lt;- paste(initialJSON, collapse="")
  </code></pre>
  
  <p>While this does work and create a character string i can read into fromJSON, this is not a scalable solution for me, since i can only read a few thousand lines at a time like this, and isn't very scalable (i'd also love to be able to stream directly from the gz file). </p>
  
  <p>Does anyone know how i can get stream_in to accept this fileformat, or some alternative way to do this using R? They show examples of how it works in Java fine, but i would love to be able to do this without jumping into a language i don't really know.</p>
  
  <h1>Update</h1>
  
  <p>Still haven't got the stream to work, but wrote my own (of sorts), seems to perform decently for my purposes.</p>
  
  <pre><code>fileCon &lt;- file('~/yasp/test.json', open="r")
  
  # Initialize everything
  numMatches &lt;- 5
  outputFile &lt;- 0
  lineCount &lt;- 0
  matchCount &lt;- 0
  matchIDList &lt;- NULL
  
  # Stream using readLines and only look at even numbered lines
  while(matchCount &lt;= numMatches) {
      next_line = readLines(fileCon, n = 1)
  
      lineCount &lt;- lineCount + 1
  
      if(lineCount %% 2 == 0) {
  
          matchCount &lt;- matchCount + 1
  
          # read into JSON format
          readJSON &lt;- jsonlite::fromJSON(next_line)
  
          # Up the match counter
          matchCount &lt;- matchCount + 1
  
          # whatever operations you want, for example get match_id
          matchIDList &lt;- c(matchIDList, readJSON$match_id)
      }
  
  }
  </code></pre>