title: How do I prevent running out of memory when inserting a million rows in mysql with php
body: <p>I have built a script in Laravel that reads a JSON file line by line and imports the contents into my database.</p>
  
  <p>However, when running the script, I get an out of memory error after inserting about 80K records.</p>
  
  <pre><code>mmap() failed: [12] Cannot allocate memory
  
  mmap() failed: [12] Cannot allocate memory
  PHP Fatal error:  Out of memory (allocated 421527552) (tried to allocate 12288 bytes) in /home/vagrant/Code/sandbox/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php on line 1758
  
  mmap() failed: [12] Cannot allocate memory
  PHP Fatal error:  Out of memory (allocated 421527552) (tried to allocate 32768 bytes) in /home/vagrant/Code/sandbox/vendor/symfony/debug/Exception/FatalErrorException.php on line 1
  </code></pre>
  
  <p>I have built a sort of makeshift queue to only commit the collected items every 100, but this made no difference.</p>
  
  <p>This is what the part of my code that does the inserts looks like:</p>
  
  <pre><code>public function callback($json) {
  
      if($json) {
  
          $this-&gt;queue[] = [
  
              'type' =&gt; serialize($json['type']),
              'properties' =&gt; serialize($json['properties']),
              'geometry' =&gt; serialize($json['geometry'])
          ];
  
          if ( count($this-&gt;queue) == $this-&gt;queueLength ) {
  
              DB::table('features')-&gt;insert( $this-&gt;queue );
  
              $this-&gt;queue = [];
          }
      }
  }
  </code></pre>
  
  <p>It's the actual inserts (<code>DB::table('features')-&gt;insert( $this-&gt;queue );</code>) that are causing the error, if I leave those out I can perfectly iterate over all lines and echo them out without any performance issues.</p>
  
  <p>I guess I could allocate more memory, but doubt this would be a solution because I'm trying to insert 3 million records and it's currently already failing after 80K with 512Mb memory allocated. Furthermore, I actually want to run this script on a low budget server.</p>
  
  <p>The time it takes for this script to run is not of any concern, so if I could somehow slow the insertion of records down that would be a solution I could settle for.</p>