title: Dealing with large curl response - PHP
body: <p>I wrote a PHP script that makes HTTP POST request using <code>curl</code> and does the following,</p>
  
  <ul>
  <li>Prepare post variables</li>
  <li>Initialize curl</li>
  <li>Set client cookie to use in request</li>
  <li>Set POST variables as query string</li>
  <li>Set other curl options</li>
  <li>Execute curl</li>
  </ul>
  
  <p>Here is the code:</p>
  
  <pre><code>    $ch = curl_init ( $url );
  
      curl_setopt ( $ch, CURLOPT_COOKIE, "cookie=cookie");
      curl_setopt ( $ch, CURLOPT_POST, 1);
      curl_setopt ( $ch, CURLOPT_POSTFIELDS, $post_string);
      curl_setopt ( $ch, CURLOPT_FOLLOWLOCATION, 1);
      curl_setopt ( $ch, CURLOPT_HEADER, 0);
      curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1);
  
      $response = curl_exec( $ch );
      // this point
      extr ( $response, $param_1, $param_2);
  </code></pre>
  
  <p>Problem is, sometimes the response is larger than 1GB, so the PHP code pauses until, full response is received (shown in code as <code>// this point</code>), and if there is malformed HTML receive, PHP generates error so, all thing here needs to do from beginning.</p>
  
  <p>Here is rest of the functions:</p>
  
  <pre><code>function extr($string = '',$a,$b)
  {
      $doc = new DOMDocument;
      @$doc -&gt; loadHTML($string);
      $table = $doc -&gt; getElementById('myTableId');
  
      if(is_object($table)):
      foreach ($table-&gt;getElementsByTagName('tr') as $record)
      {
          $rec = array();
          foreach ($record-&gt;getElementsByTagName('td') as $data)
          {
              $rec[] = $data -&gt; nodeValue;
          }
          if ($rec)
          {
              put_data($rec);
          }
      }
      else:
      {
          echo 'Skipped: Param1:'.$a.'-- Param2: '.$b.'&lt;br&gt;';
      }
      endif;
  }
  
  function put_data($one = array())
  {
      $one = json_encode($one) . "\n";
      file_put_contents("data.json", $one, FILE_APPEND);
  }
  
  ini_set('max_execution_time', 3000000);
  ini_set('memory_limit', '-1');
  </code></pre>
  
  <p>The alternative i can think of is process data as it received, if possible , using curl, or continue previous curl request from the previous state.</p>
  
  <p>Is there any possible workaround for this?</p>
  
  <p>Do i need to switch to any other language than PHP for this ?</p>