title: Scrapy - parse a page to extract items - then follow and store item url contents
body: <p>I have a question on how to do this thing in scrapy. I have a spider that crawls for listing pages of items. 
  Every time a listing page is found, with items, there's the parse_item() callback that is called for extracting items data, and yielding items. So far so good, everything works great.</p>
  
  <p>But each item, has among other data, an url, with more details on that item. I want to follow that url and store in another item field (url_contents) the fetched contents of that item's url. </p>
  
  <p>And I'm not sure how to organize code to achieve that, since the two links (listings link, and one particular item link) are followed differently, with callbacks called at different times, but I have to correlate them in the same item processing.</p>
  
  <p>My code so far looks like this:</p>
  
  <pre><code>class MySpider(CrawlSpider):
      name = "example.com"
      allowed_domains = ["example.com"]
      start_urls = [
          "http://www.example.com/?q=example",
      ]
  
      rules = (
          Rule(SgmlLinkExtractor(allow=('example\.com', 'start='), deny=('sort='), restrict_xpaths = '//div[@class="pagination"]'), callback='parse_item'),
          Rule(SgmlLinkExtractor(allow=('item\/detail', )), follow = False),
      )
  
  
      def parse_item(self, response):
          main_selector = HtmlXPathSelector(response)
          xpath = '//h2[@class="title"]'
  
          sub_selectors = main_selector.select(xpath)
  
          for sel in sub_selectors:
              item = ExampleItem()
              l = ExampleLoader(item = item, selector = sel)
              l.add_xpath('title', 'a[@title]/@title')
              ......
              yield l.load_item()
  </code></pre>