title: Memory issue retrieving many records from Dynamics CRM using SDK query with paging
body: <p>I'm trying to set up a basic web service (.NET 4.5.2 hosted on IIS 7.5) to query some data in Dynamics CRM using the SDK (we're using CRM 2015 online).  Depending on the query there could be a lot of records to return, for example querying the Account entity I might expect to get back as much as 250000 records.  I'm using paging to get the results and stuff them into a final EntityCollection object so I can work on the full data set once all the records are returned.  The problem is I'm getting an <code>OutOfMemory</code> exception usually before it finishes querying all pages.  Here's a code snippet showing the While loop I'm working with:</p>
  
  <pre><code>While lblnMoreRecords
     lEC = _CRMHelper.OrgService.RetrieveMultiple(New FetchExpression(lstrXml))
     lblnMoreRecords = lEC.MoreRecords
  
     If lEC.Entities.Count &gt; 0 Then
        pecEntities.Entities.AddRange(lEC.Entities)
        lintPage += 1
        lstrXml = SetFetchXMLPaging(pstrFetchXML, lEC.PagingCookie, lintPage, 200)
     Else
        lblnMoreRecords = False
     End If
  
     lEC = Nothing
  End While
  </code></pre>
  
  <p>In this snippet I'm using FetchXML to get the results, although I have also tried the same sort of loop with a QueryExpression object and paging.  I can run this process and watch it's memory usage as it's running, the memory consumption comes in at the line <code>lEC = _CRMHelper.OrgService.RetrieveMultiple(New FetchExpression(lstrXml))</code>, where it is doing the actual query and returning the results from CRM.  The process will occasionally free up some memory but overall it increases until I get the exception.  Besides trying a QueryExpression, I also tried reducing the number of records per page from 1000 to 200, hoping that by using less memory for each iteration would mean the OS could more easily allocate free blocks of memory to the process.  That helped but I still get the exception before all results are returned.</p>
  
  <p>One more thing I tried was to run this query in a basic Windows App rather than a web service. The windows app seemed to be much better at managing memory, it got to a certain point and stayed steady, easily finishing the query and getting all the results.</p>
  
  <p>Any suggestions on dealing with this issue in a web service/IIS would be greatly appreciated, or does anyone know of a better way to query CRM via their SDK to get large sets of data?  Thanks.</p>