title: Scrapy Result not being written
body: <p>I'm scraping the following site: <a href="https://graphics.stltoday.com/apps/payrolls/salaries/teachers/" rel="nofollow noreferrer">https://graphics.stltoday.com/apps/payrolls/salaries/teachers/</a></p>
  
  <p>Hoping to scrape all the data for each individual. This means following the link to each district, then to each job category within the district, and finally to each employee. I think the issue may be with my regular expressions for the URLs, but I'm not sure. Once on each employee's page, I think I've correctly identified the XPaths:</p>
  
  <pre><code>import scrapy
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  
  class Spider2(CrawlSpider):
      #name of the spider
      name = 'stltoday'
  
      #list of allowed domains
      allowed_domains = ['graphics.stltoday.com']
  
      #starting url for scraping
      start_urls = ['https://graphics.stltoday.com/apps/payrolls/salaries/teachers']
  
      rules = [
      Rule(LinkExtractor(
          allow=['/[0-9]+/$']),
          follow=True),
      Rule(LinkExtractor(
          allow=['/[0-9]+/position/[0-9]+/$']),
          follow=True),
      Rule(LinkExtractor(
          allow=['/detail/[0-9]+/$']),
          callback='parse_item',
          follow=True),
      ]
  
      #setting the location of the output csv file
      custom_settings = {
          'FEED_FORMAT' : "csv",
          'FEED_URI' : 'tmp/stltoday1.csv'
      }
  
      def parse_item(self, response):
          #Remove XML namespaces
          response.selector.remove_namespaces()
          url = response.url
          #Extract article information
  
          fullname = response.xpath('//p[@class="table__title"]./text()').extract_first()
  
          for row in response.xpath('//th[@scope="row"]'):
              yield {
              "url": url,
          "fullname": fullname,
              "district": row.xpath('./text()').extract_first(),
              "school": row.xpath('./following-sibling::*[1]/text()').extract_first(),
              "degree": row.xpath('./following-sibling::*[2]/text()').extract_first(),
              "salary": row.xpath('./following-sibling::*[3]/text()').extract_first(),
          "extcontractpay": row.xpath('./following-sibling::*[4]/text()').extract_first(),
          "extraduty": row.xpath('./following-sibling::*[5]/text()').extract_first(),
          "totalpay": row.xpath('./following-sibling::*[6]/text()').extract_first(),
          "yearsindistrict": row.xpath('./following-sibling::*[7]/text()').extract_first(),
          "yearsinmoschools": row.xpath('./following-sibling::*[8]/text()').extract_first(),
              }
  
  
          for item in zip(url,fullname,district,school,degree,salary,extcontractpay,extraduty,totalpay,yearsindistrict,yearsinmoschools):
              yield {
                  'url' : url,
          'fullname' : fullname,
                  'district' : district,
                  'school' : school,
                  'degree' : degree,
                  'salary' : salary,
          'extcontractpay' : extcontractpay,
                  'extraduty' : extraduty,
                  'totalpay' : totalpay,
                  'yearsindistrict' : yearsindistrict,
                  'yearsinmoschools' : yearsinmoschools
              }
  </code></pre>
  
  <p>The spider runs (for a few minutes before I pause it), but nothing is written to the .csv file.     </p>