title: Using scrapy to loop through discovered a-href url links to scrape the corresponding pages
body: <p>I'm just starting out with Scrapy and Python, and have been following the tutorial, but am stuck. I have been able to use the shell to get a list of links from a page, as per the below: </p>
  
  <pre><code>&gt;&gt;&gt; response.css('li').xpath('a/@href').getall()
  </code></pre>
  
  <p>Gives me: </p>
  
  <pre><code>'/shop-online/542/fragrances', '/shop-online/81/vitamins', '/shop-online/257/beauty', '/shop-online/665/skin-care', '/shop-online/648/cosmetics', '/shop-online/517/weight-loss', '/shop-online/20/baby-care', '/shop-online/89/sexual-health', '/shop-online/198/smoking-deterrents', '/shop-online/3240/clearance', '/prescriptions', '/shop-online/258/medicines', '/shop-online/1093/cold-flu', '/shop-online/PS-1755/all-fish-oil-supplements', '/shop-online/159/oral-hygiene-and-dental-care', '/shop-online/792/household', '/shop-online/129/hair-care', '/shop-online/1255/sports-nutrition', '/bestsellers', '/categories', 'https://www.chemistwarehouse.hk', '/', '#', '/login', '/youraccount', '#', '/aboutus', '/aboutus/shipping', '/shop-online/542/fragrances', '/shop-online/81/vitamins', '/shop-online/257/beauty', '/shop-online/665/skin-care', '/shop-online/648/cosmetics', '/shop-online/517/weight-loss', '/shop-online/20/baby-care', '/shop-online/89/sexual-health', '/shop-online/198/smoking-deterrents', '/prescriptions', '/shop-online/258/medicines', '/shop-online/1093/cold-flu', '/shop-online/PS-1755/all-fish-oil-supplements', '/shop-online/159/oral-hygiene-and-dental-care', '/shop-online/792/household', '/shop-online/129/hair-care', '/shop-online/1255/sports-nutrition', '/bestsellers']
  </code></pre>
  
  <p>What I want to be able to do, at least using the shell for now (and then scripting it) is to be able to parse out any links that don't contain shop-online, and then scrape the corresponding URLs, which will be www..website/<strong>the link I've scraped</strong></p>
  
  <p>But I'm not sure how to do this. I am aware there are regex expressions you can use, but I'm not sure how to apply them, and even if I could, I'm not sure how to then tell scrapy to iterate through what I've found and scrape THOSE pages? </p>