title: Mimicking an ajax call with Curl PHP
body: <p>I'm scraping a site using curl (via PHP) and some information I want is a list of products which is by default only showing the first few ones. The rest is passed to the user when they click a button to get the full list of products, which triggers an ajax call to return that list.</p>
  
  <p>Here is in a nutshell the JS they use:</p>
  
  <pre><code>headers['__RequestVerificationToken'] = token;
  $.ajax({
  type: "post",
  url: "/ajax/getProductList",
  dataType: 'html',
  data: JSON.stringify({ historyPageIndex: 1, displayPeriod: 0, productsType: All }),
  contentType: 'application/json; charset=utf-8',
  success: function (result) {
      $(target).html("");
      $(target).html(result);
  },
  beforeSend: function (XMLHttpRequest) {
      if (headers['__RequestVerificationToken']) {
          XMLHttpRequest.setRequestHeader("__RequestVerificationToken", headers['__RequestVerificationToken']);
      }
  }
  });
  </code></pre>
  
  <p>Here is my PHP script:</p>
  
  <pre><code>curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
  curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieLocation);
  curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieLocation);
  curl_setopt($ch, CURLOPT_POST, false);
  curl_setopt($ch, CURLOPT_URL, 'https://www.domain.com/Applications/ViewProducts');
  curl_setopt($ch, CURLOPT_REFERER, 'https://www.domain.com/');
  $webpage = curl_exec($ch);
  $productsType = trim(find_by_pattren($webpage, '&lt;input id="productsType" name="productsType" type="hidden" value="(.*?)"'));
  $token = trim(find_by_pattren($webpage, '&lt;input name="__RequestVerificationToken" type="hidden" value="(.*?)"'));
  
  $postVariables = 'productsType='.$productsType.
  '&amp;historyPageIndex=1
  &amp;displayPeriod=0
  &amp;__RequestVerificationToken='.$token;
  curl_setopt($ch, CURLOPT_POST, true);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $postVariables);
  curl_setopt($ch, CURLOPT_URL, 'https://www.domain.com/ajax/getProductList');
  curl_setopt($ch, CURLOPT_REFERER, 'https://www.domain.com/Applications/ViewProducts');
  $webpage = curl_exec($ch);
  </code></pre>
  
  <p>This produces an error page with the site. I think the main reasons could be that:</p>
  
  <ul>
  <li><p>They check whether it's an ajax request (no clue how to fix that)</p></li>
  <li><p>The token needs to be in the header and not in the post variables</p></li>
  </ul>
  
  <p>Any idea?</p>
  
  <p>EDIT: here is the working code:</p>
  
  <pre><code>curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
  curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieLocation);
  curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieLocation);
  curl_setopt($ch, CURLOPT_URL, 'https://www.domain.com/Applications/ViewProducts');
  curl_setopt($ch, CURLOPT_REFERER, 'https://www.domain.com/');
  $webpage = curl_exec($ch);
  $productsType = trim(find_by_pattren($webpage, '&lt;input id="productsType" name="productsType" type="hidden" value="(.*?)"'));
  $token = trim(find_by_pattren($webpage, '&lt;input name="__RequestVerificationToken" type="hidden" value="(.*?)"'));
  
  $postVariables = json_encode(array('productsType' =&gt; $productsType,
  'historyPageIndex' =&gt; 1,
  'displayPeriod' =&gt; 0));
  curl_setopt($ch, CURLOPT_POST, true);
  curl_setopt($ch, CURLOPT_HTTPHEADER, array("X-Requested-With: XMLHttpRequest", "Content-Type: application/json; charset=utf-8", "__RequestVerificationToken: $token"));
  curl_setopt($ch, CURLOPT_POSTFIELDS, $postVariables);
  curl_setopt($ch, CURLOPT_URL, 'https://www.domain.com/ajax/getProductList');
  curl_setopt($ch, CURLOPT_REFERER, 'https://www.domain.com/Applications/ViewProducts');
  $webpage = curl_exec($ch);
  </code></pre>