title: How do Scrapy rules work with crawl spider
body: <p>I have hard time to understand scrapy crawl spider rules. I have example that doesn't work as I would like it did, so it can be two things:</p>
  
  <ol>
  <li>I don't understand how rules work.</li>
  <li>I formed incorrect regex that prevents me to get results that I need.</li>
  </ol>
  
  <p>OK here it is what I want to do:</p>
  
  <p>I want to write crawl spider that will get all available statistics information from <a href="http://www.euroleague.net" rel="noreferrer">http://www.euroleague.net</a> website.
  The website page that hosts all the information that I need for the start is <a href="http://www.euroleague.net/main/results/by-date" rel="noreferrer">here</a>.</p>
  
  <p><strong>Step 1</strong></p>
  
  <p>First step what I am thinking is extract "Seasons" link(s) and fallow it.
  Here it is HTML/href that I am intending to match (I want to match all links in the "Seasons" section one by one, but I think that it will be easer to have one link as an example):</p>
  
  <pre><code>href="/main/results/by-date?seasoncode=E2001"
  </code></pre>
  
  <p>And here is a rule/regex that I created for it:</p>
  
  <pre><code>Rule(SgmlLinkExtractor(allow=('by-date\?seasoncode\=E\d+',)),follow=True),
  </code></pre>
  
  <p><img src="https://i.stack.imgur.com/AhJ2M.png" alt="enter image description here"></p>
  
  <p><strong>Step 2</strong></p>
  
  <p>When I am brought by spider to the web page <a href="http://www.euroleague.net/main/results/by-date?seasoncode=E2001" rel="noreferrer">http://www.euroleague.net/main/results/by-date?seasoncode=E2001</a> for the second step I want that spider extracted link(s) from section "Regular season". At this case lets say it should be "Round 1". The HTML/href that I am looking for is:</p>
  
  <pre><code>&lt;a href="/main/results/by-date?seasoncode=E2001&amp;gamenumber=1&amp;phasetypecode=RS"
  </code></pre>
  
  <p>And rule/regex that I constructed would be:</p>
  
  <pre><code>Rule(SgmlLinkExtractor(allow=('seasoncode\=E\d+\&amp;gamenumber\=\d+\&amp;phasetypecode\=\w+',)),follow=True),
  </code></pre>
  
  <p><img src="https://i.stack.imgur.com/6pLhL.png" alt="enter image description here"></p>
  
  <p><strong>Step 3</strong></p>
  
  <p>Now I reached page (<a href="http://www.euroleague.net/main/results/by-date?seasoncode=E2001&amp;gamenumber=1&amp;phasetypecode=RS" rel="noreferrer">http://www.euroleague.net/main/results/by-date?seasoncode=E2001&amp;gamenumber=1&amp;phasetypecode=RS</a>) I am ready to extract links that leads to the pages that has all the information that I need:
  I am looking for HTML/href:</p>
  
  <pre><code>href="/main/results/showgame?gamenumber=1&amp;phasetypecode=RS&amp;gamecode=4&amp;seasoncode=E2001#!boxscore"
  </code></pre>
  
  <p>And my regex that has to follow would be:</p>
  
  <pre><code>Rule(SgmlLinkExtractor(allow=('gamenumber\=\d+\&amp;phasetypecode\=\w+\&amp;gamecode\=\d+\&amp;seasoncode\=E\d+',)),callback='parse_item'),
  </code></pre>
  
  <p><img src="https://i.stack.imgur.com/1qF7o.png" alt="enter image description here"></p>
  
  <p><strong>The problem</strong></p>
  
  <p>I think that crawler should work something like this:
  That rules crawler is something like a loop. When first link is matched the crawler will follow to the "Step 2" page, than to "step 3" and after that it will extract data. After doing that it will return to "step 1" to match second link and start loop again to the point when there is no links in first step.</p>
  
  <p>What I see from terminal it seems that crawler loops in "Step 1". It loops through all "Step 1" links, but doesn't involves "step 2"/"step 3" rules. </p>
  
  <pre><code>2014-02-28 00:20:31+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2000&gt; (referer: http://  www.euroleague.net/main/results/by-date)
  2014-02-28 00:20:31+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2001&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 00:20:31+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2002&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 00:20:32+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2003&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 00:20:33+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2004&gt; (referer: http://www.euroleague.net/main/results/by-date)
  </code></pre>
  
  <p>After it loops through all the "Seasons" links it starts with links that I don't see, in any of three steps that I mentioned:</p>
  
  <pre><code>http://www.euroleague.net/main/results/by-date?gamenumber=23&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013
  </code></pre>
  
  <p>And such link structure you can find only if you loop through all the links in "Step 2" without returning to the "Step 1" starting point. </p>
  
  <p><strong>The question would be:</strong>
  How rules work? Is it working step by step like I am intending it should work with this example or every rule has it's own loop and goes from rule to rule only after it's finished looping through the first rule?</p>
  
  <p>That is how I see it. Of course it could be something wrong with my rules/regex and it is very possible.</p>
  
  <p>And here is all what I am getting from the terminal:</p>
  
  <pre><code>scrapy crawl basketsp_test -o item6.xml -t xml
  2014-02-28 01:09:20+0200 [scrapy] INFO: Scrapy 0.20.0 started (bot: basketbase)
  2014-02-28 01:09:20+0200 [scrapy] DEBUG: Optional features available: ssl, http11, boto, django
  2014-02-28 01:09:20+0200 [scrapy] DEBUG: Overridden settings: {'NEWSPIDER_MODULE': 'basketbase.spiders', 'FEED_FORMAT': 'xml', 'SPIDER_MODULES': ['basketbase.spiders'], 'FEED_URI': 'item6.xml', 'BOT_NAME': 'basketbase'}
  2014-02-28 01:09:21+0200 [scrapy] DEBUG: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
  2014-02-28 01:09:21+0200 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
  2014-02-28 01:09:21+0200 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
  2014-02-28 01:09:21+0200 [scrapy] DEBUG: Enabled item pipelines: Basketpipeline3, Basketpipeline1db
  2014-02-28 01:09:21+0200 [basketsp_test] INFO: Spider opened
  2014-02-28 01:09:21+0200 [basketsp_test] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2014-02-28 01:09:21+0200 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
  2014-02-28 01:09:21+0200 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
  2014-02-28 01:09:21+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date&gt; (referer: None)
  2014-02-28 01:09:22+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:22+0200 [basketsp_test] DEBUG: Filtered duplicate request: &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2013&gt; - no more duplicates will be shown (see DUPEFILTER_CLASS)
  2014-02-28 01:09:22+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2000&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:23+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2001&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:23+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2002&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:24+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2003&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:24+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2004&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:25+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2005&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:26+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2006&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:26+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2007&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:27+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2008&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:27+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2009&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:28+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2010&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:29+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2011&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:29+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?seasoncode=E2012&gt; (referer: http://www.euroleague.net/main/results/by-date)
  2014-02-28 01:09:30+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=24&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:30+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=23&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:31+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=22&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:32+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=21&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:32+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=20&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:33+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=19&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:34+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=18&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:34+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=17&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:35+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=16&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:35+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=15&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:36+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=14&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:37+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=13&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:37+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=12&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:38+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=11&amp;phasetypecode=TS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:39+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=10&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:39+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=9&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:40+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=8&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:40+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=7&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:41+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=6&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:42+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=5&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:42+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=4&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:43+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=3&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:44+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=2&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:44+0200 [basketsp_test] DEBUG: Crawled (200) &lt;GET http://www.euroleague.net/main/results/by-date?gamenumber=1&amp;phasetypecode=RS++++++++&amp;seasoncode=E2013&gt; (referer: http://www.euroleague.net/main/results/by-date?seasoncode=E2013)
  2014-02-28 01:09:44+0200 [basketsp_test] INFO: Closing spider (finished)
  2014-02-28 01:09:44+0200 [basketsp_test] INFO: Dumping Scrapy stats:
      {'downloader/request_bytes': 13663,
       'downloader/request_count': 39,
       'downloader/request_method_count/GET': 39,
       'downloader/response_bytes': 527838,
       'downloader/response_count': 39,
       'downloader/response_status_count/200': 39,
       'finish_reason': 'finished',
       'finish_time': datetime.datetime(2014, 2, 27, 23, 9, 44, 569579),
       'log_count/DEBUG': 46,
       'log_count/INFO': 3,
       'request_depth_max': 2,
       'response_received_count': 39,
       'scheduler/dequeued': 39,
       'scheduler/dequeued/memory': 39,
       'scheduler/enqueued': 39,
       'scheduler/enqueued/memory': 39,
       'start_time': datetime.datetime(2014, 2, 27, 23, 9, 21, 111255)}
  2014-02-28 01:09:44+0200 [basketsp_test] INFO: Spider closed (finished)
  </code></pre>
  
  <p>And here is a rules part from the crawler:</p>
  
  <pre><code>class Basketspider(CrawlSpider):
      name = "basketsp_test"
      download_delay = 0.5
  
      allowed_domains = ["www.euroleague.net"]
      start_urls = ["http://www.euroleague.net/main/results/by-date"]
      rules = (
          Rule(SgmlLinkExtractor(allow=('by-date\?seasoncode\=E\d+',)),follow=True),
          Rule(SgmlLinkExtractor(allow=('seasoncode\=E\d+\&amp;gamenumber\=\d+\&amp;phasetypecode\=\w+',)),follow=True),
          Rule(SgmlLinkExtractor(allow=('gamenumber\=\d+\&amp;phasetypecode\=\w+\&amp;gamecode\=\d+\&amp;seasoncode\=E\d+',)),callback='parse_item'),
  
  
  
  )  
  </code></pre>