title: How to monitor health of Scrapy spiders
body: <p>I have a Scrapy project that parses around 30+ listing sites, and I am trying to figure out a way on how to maintain and monitor the consistency of the scrapers should the underlaying web structure of any of the sites be updated. </p>
  
  <p>There are two ways already that I use to try and achieve this:</p>
  
  <ol>
  <li><p>I have a set of rules that <code>DropItem</code> if a few core fields are not present (product title, price, and couple others that are always present). This helps somewhat, but it assumes that if those 3/4 fields are ok then the rest of the structure is ok too. For context, I can't make all other fields required as they by nature are optional (i.e <code>colour</code>, <code>size</code>, etc)</p></li>
  <li><p>I have unitests that work by running the scrapers on a set of locally saved HTML bodies for each spider. This is good for overall code quality, but does not solve the problem of determining that a spider might be broken because of a DOM change. </p></li>
  </ol>
  
  <p>There are a few things that I thought of doing along the lines of;</p>
  
  <ul>
  <li>Potentially calculating an average of how often a field is populated per crawl. If <code>size</code> is present on average <code>30%</code> of the time, a job that is returning <code>90%</code> or <code>1%</code> could raise an alert. However I can think of many circumstances where either this would trigger a number of false-positives, or even not trigger a thing and still be broken</li>
  <li>Another thought I had was having a service that somehow monitors the HTML structure, this could be ran before a scrape job and pause if a change is detected. To me this seems like the option that would be the most resilient, but I have no idea how to achieve that. </li>
  </ul>
  
  <p><strong>TL;DR</strong> </p>
  
  <p>How can I monitor for changes in the structure of a HTML page so that I can avoid running Scrapy jobs that result in corrupted data? </p>
  
  <p>What other options are there to detect that scraping jobs might be corrupted, heuristic or otherwise? <code>DropItem</code> already being used wherever possible. </p>