title: Beautiful Soup open all the url with pid in it
body: <p>I am trying to open all the links through pid in it but there two situations:</p>
  
  <ol>
  <li><p>Where it opens all the url ( I mean even the junk urls)</p>
  
  <pre><code>def get_links(self): 
      links = [] 
      host = urlparse( self.url ).hostname 
      scheme = urlparse( self.url ).scheme 
      domain_link = scheme+'://'+host 
      pattern = re.compile(r'(/pid/)')
  
      for a in self.soup.find_all(href=True):            
          href = a['href']
          if not href or len(href) &lt;= 1:
              continue
          elif 'javascript:' in href.lower():
              continue
          elif 'forgotpassword' in href.lower():
              continue
          elif 'images' in href.lower():
              continue
          elif 'seller-account' in href.lower():
              continue
          elif 'review' in href.lower():
              continue
          else:
              href = href.strip()
          if href[0] == '/':
              href = (domain_link + href).strip()
          elif href[:4] == 'http':
              href = href.strip()
          elif href[0] != '/' and href[:4] != 'http':
              href = ( domain_link + '/' + href ).strip()                  
          if '#' in href:
              indx = href.index('#')
              href = href[:indx].strip()
          if href in links:
              continue
  
          links.append(self.re_encode(href))
  
      return links
  </code></pre></li>
  <li><p>In this case it just opens the url with pid in it but in this case it does not follow links and only limited to the homepage. After opening few links with pid it crashes.</p>
  
  <pre><code>def get_links(self): 
      links = [] 
      host = urlparse( self.url ).hostname 
      scheme = urlparse( self.url ).scheme 
      domain_link = scheme+'://'+host 
      pattern = re.compile(r'(/pid/)')
  
      for a in self.soup.find_all(href=True):
          if pattern.search(a['href']) is not None:
             href = a['href']  
              if not href or len(href) &lt;= 1:
                  continue
              elif 'javascript:' in href.lower():
                  continue
              elif 'forgotpassword' in href.lower():
                  continue
              elif 'images' in href.lower():
                  continue
              elif 'seller-account' in href.lower():
                  continue
              elif 'review' in href.lower():
                  continue
              else:
                  href= href.strip()
              if href[0] == '/':
                  href = (domain_link + href).strip()
              elif href[:4] == 'http':
                  href = href.strip()
              elif href[0] != '/' and href[:4] != 'http':
                  href = ( domain_link + '/' + href ).strip()                  
              if '#' in href:
                 indx = href.index('#')
                 href = href[:indx].strip()
              if href in links:
                 continue
  
              links.append(self.re_encode(href))
  
      return links
  </code></pre></li>
  </ol>
  
  <p>Can someone help to get all the links even the internal links within the url and at the end only accept the pid as the returned link.</p>