title: Scrapy: Following Links with regular Expression
body: <p>i want to crawl threads from a german Forum. <a href="http://www.musiker-board.de/" rel="nofollow">http://www.musiker-board.de/</a></p>
  
  <p>The actual different subforums are located at <a href="http://www.musiker-board.de/forum" rel="nofollow">http://www.musiker-board.de/forum</a></p>
  
  <p>A subforum: musiker-board.de/forum/subforumname</p>
  
  <p>The actual Threads have these adresses: musiker-board.de/threads/threadname</p>
  
  <p>I want to follow all links of all subforums and extract all threads in them,
  but the URL of the threads wouldnt match with the start URL anymore.</p>
  
  <p>However if I select "musiker-board.de/" as a start URL it doesnt follow the links of all subforums.</p>
  
  <p>Here is the code:    </p>
  
  <pre><code>allowed_domains = ["musiker-board.de"]
  start_urls = ['http://www.musiker-board.de/forum/'
               ]
  rules = (
           Rule(SgmlLinkExtractor(allow=[r'forum/\w+']), follow=True),
           Rule(SgmlLinkExtractor(allow=[r'threads/\w+']), callback='parse_item'),
           )
  
  def parse_item(self, response):
      #extract items...
  </code></pre>
  
  <p>What should i do to follow all musiker-board.de/forum/subforum and to extract all musiker-forum.de/threads/threadname ?</p>