title: HtmlAgilityPack & Selenium Webdriver returns random results
body: <p>I'm trying to scrape product names from a website.  Oddly, I seem to only scrape random 12 items.  I've tried both HtmlAgilityPack and with HTTPClient and I get the same random results.  Here's my code for HtmlAgilityPack:</p>
  
  <pre><code>using HtmlAgilityPack;
  using System.Net.Http;
  
  var url = @"http://www.roots.com/ca/en/men/tops/shirts-and-polos/";
  HtmlWeb web = new HtmlWeb();
  var doc = web.Load(url, "GET", proxy, new NetworkCredential(PROXY_UID, PROXY_PWD, PROXY_DMN));
  var nodes = doc.DocumentNode.Descendants("div")
              .Where(div =&gt; div.GetAttributeValue("class", string.Empty) == "product-name")
              .Select(div =&gt; div.InnerText.Trim())
              ;
  </code></pre>
  
  <p><strong>[UPDATE 1]</strong>
  @CodingKuma suggested I try Selenium Webdriver.  Here's my code using Selenium Webdriver:</p>
  
  <pre><code>IWebDriver chromeDriver = new ChromeDriver(@"C:\TEMP\Projects\Chrome\chromedriver_win32");
  chromeDriver.Url = "http://www.roots.com/ca/en/men/tops/shirts-and-polos/";
  var items = chromeDriver.FindElements(By.ClassName("product-name"));
  items.Count().Dump();
  chromeDriver.Quit();
  </code></pre>
  
  <p>I tried this code but still no luck.  There are over 20 items on that page, but I seem to only get a random 12.  How can I scrape all items on that site?</p>