title: How to test a web scraping application?
body: <p>I am using <a href="http://nokogiri.org/" rel="nofollow">Nokogiri</a> to check that an automatically generated site has the correct information and sub sites. It works, but I need to test it, and I'm not sure of a good way to do so. My current approach is to find a site online that I know has certain information, and make sure that the application finds that.</p>
  
  <p>So, for example, I visually check to see how many rows there are in a table on the site, and then test to see if the function that checks that table returns that number of rows. Both the URL and the number are hardcoded. The site will probably not change, but it is not impossible either, in which case the tests will obviously fail. </p>
  
  <p>Is there any better way to do this?</p>