title: TypeError: 'HtmlResponse' object is not iterable
body: <p>I'm new to python, but trying to get my head around it in order to use Scrapy for work.</p>
  
  <p>I'm currently following this tutorial:
  <a href="http://scrapy2.readthedocs.io/en/latest/intro/tutorial.html" rel="nofollow">http://scrapy2.readthedocs.io/en/latest/intro/tutorial.html</a></p>
  
  <p>I've having trouble with this part (from the tutorial): </p>
  
  <pre><code>def parse(self, response):
      for sel in response.xpath('//ul/li'):
          title = sel.xpath('a/text()').extract()
          link = sel.xpath('a/@href').extract()
          desc = sel.xpath('text()').extract()
          print title, link, desc
  </code></pre>
  
  <p>The problem I'm having is with the <code>for sel in response.xpath('//ul/li'):</code> part. I understand that line to essentially narrow down what is crawled to anything that matches the xpath <code>//ul/li</code>.</p>
  
  <p>However, in my implementation, I can't narrow down the page to one singular section. I tried to get around this by selecting the entire HTML, see my attempt below:</p>
  
  <pre><code>   def parse(self, response):
             for sel in response.xpath('//html'):
              title = sel.xpath('//h1/text()').extract()
              author = sel.xpath('//head/meta[@name="author"]/@content').extract()
              mediumlink = sel.xpath('//head/link[@rel="author"]/@href').extract()
              print title, author, mediumlink
  </code></pre>
  
  <p>The xpaths work both in a Chrome plugin I use, and using <code>response.xpath('//title').extract()</code> in <code>scrapy shell</code></p>
  
  <p>I've tried changing the line to this:</p>
  
  <p><code>for sel in response.xpath('//html'):</code> and <code>for sel in response.xpath('html'):</code></p>
  
  <p>But each time I get this:</p>
  
  <pre><code>2016-10-16 14:33:43 [scrapy] ERROR: Spider error processing &lt;GET https://medium.com/swlh/how-our-app-went-from-20-000-day-to-2-day-in-revenue-d6892a2801bf#.smmwwqxlf&gt; (referer: None)
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 587, in _runCallbacks
      current.result = callback(current.result, *args, **kw)
    File "/Users/Matthew/Sites/crawl/tutorial/tutorial/spiders/medium_Spider.py", line 11, in parse
      for sel in response:
  TypeError: 'HtmlResponse' object is not iterable
  </code></pre>
  
  <p>Could somebody give me some pointers on how best to resolve this? Go easy on me, my skills are not so hot. Thanks!</p>