title: Recursively parse all category links and get all products
body: <p>I've been playing around with web-scraping (<em>for this practice exercise using Python 3.6.2</em>) and I feel like I'm loosing it a bit. Given <a href="https://www.richelieu.com/us/en/category/custom-made-cabinet-doors-and-drawers/1000128" rel="nofollow noreferrer">this</a> example link, here's what I want to do:</p>
  <p>First, as you can see, there are multiple categories on the page. Clicking each of the categories from above will give me other categories, then other ones, an so on, until I reach the products page. So I have to go in depth <em>x</em> number of times. I thought recursion will help me achieve this, but somewhere I did something wrong.</p>
  <h3>Code:</h3>
  <p>Here, I'll explain the way I approached the problem. First, I created a session and a simple generic function which will return a <code>lxml.html.HtmlElement</code> object:</p>
  <pre><code>from lxml import html
  from requests import Session
  
  
  HEADERS = {
      &quot;User-Agent&quot;: &quot;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) &quot;
                    &quot;Chrome/62.0.3202.94 Safari/537.36&quot;
  }
  TEST_LINK = 'https://www.richelieu.com/us/en/category/custom-made-cabinet-doors-and-drawers/1000128'
  
  session_ = Session()
  
  
  def get_page(url):
      page = session_.get(url, headers=HEADERS).text
      return html.fromstring(page)
  </code></pre>
  <p>Then, I thought I'll need two other functions:</p>
  <ul>
  <li>one to get the category links</li>
  <li>and another one to get the product links</li>
  </ul>
  <p>To distinguish between one and another, I figured out that only on category pages, there's a title which contains <code>CATEGORIES</code> every time, so I used that:</p>
  <pre><code>def read_categories(page):
      categs = []
      try:
          if 'CATEGORIES' in page.xpath('//div[@class=&quot;boxData&quot;][2]/h2')[0].text.strip():
              for a in page.xpath('//*[@id=&quot;carouselSegment2b&quot;]//li//a'):
                  categs.append(a.attrib[&quot;href&quot;])
              return categs
          else:
              return None
      except Exception:
          return None
  
  
  def read_products(page):
      return [
          a_tag.attrib[&quot;href&quot;]
          for a_tag in page.xpath(&quot;//ul[@id='prodResult']/li//div[@class='imgWrapper']/a&quot;)
      ]
  </code></pre>
  <p>Now, the only thing left, is the recursion part, where I'm sure I did something wrong:</p>
  <pre><code>def read_all_categories(page):
      cat = read_categories(page)
      if not cat:
          yield read_products(page)
      else:
          yield from read_all_categories(page)
  
  
  def main():
      main_page = get_page(TEST_LINK)
  
      for links in read_all_categories(main_page):
          print(links)
  </code></pre>
  <p>Here's all the code put together:</p>
  <pre><code>from lxml import html
  from requests import Session
  
  
  HEADERS = {
      &quot;User-Agent&quot;: &quot;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) &quot;
                    &quot;Chrome/62.0.3202.94 Safari/537.36&quot;
  }
  TEST_LINK = 'https://www.richelieu.com/us/en/category/custom-made-cabinet-doors-and-drawers/1000128'
  
  session_ = Session()
  
  
  def get_page(url):
      page = session_.get(url, headers=HEADERS).text
      return html.fromstring(page)
  
  
  def read_categories(page):
      categs = []
      try:
          if 'CATEGORIES' in page.xpath('//div[@class=&quot;boxData&quot;][2]/h2')[0].text.strip():
              for a in page.xpath('//*[@id=&quot;carouselSegment2b&quot;]//li//a'):
                  categs.append(a.attrib[&quot;href&quot;])
              return categs
          else:
              return None
      except Exception:
          return None
  
  
  def read_products(page):
      return [
          a_tag.attrib[&quot;href&quot;]
          for a_tag in page.xpath(&quot;//ul[@id='prodResult']/li//div[@class='imgWrapper']/a&quot;)
      ]
  
  
  def read_all_categories(page):
      cat = read_categories(page)
      if not cat:
          yield read_products(page)
      else:
          yield from read_all_categories(page)
  
  
  def main():
      main_page = get_page(TEST_LINK)
  
      for links in read_all_categories(main_page):
          print(links)
  
  
  if __name__ == '__main__':
      main()
  </code></pre>
  <hr />
  <p>Could someone please point me into the right direction regarding the recursion function?</p>