title: How to exclude certain contents from being web scraped by Python
body: <p>I am trying to web scrape English questions with Python from a website (I already obtained the permission for doing this beforehand); I am using <code>BeautifulSoup</code>.</p>
  <p>The English questions are nested between the tags <code>&lt;div class=&quot;question_body&quot;&gt;</code> and <code>&lt;/div&gt;</code>. So below is my Python code that I wrote to extract all the English questions:</p>
  <pre class="lang-py prettyprint-override"><code>import requests
  import pandas as pd
  from bs4 import BeautifulSoup
  
  for p in range(1,10):
      web_page = requests.get('https://www.helpteaching.com/search/index.htm?grade=90&amp;question_type=1&amp;keyword=&amp;entity=7&amp;pageNum={}'.format(p))
     
      # Parse web_page
      soup = BeautifulSoup(web_page.text, 'html.parser')
      
      # Create set of results based on HTML tags with desired data
      results = soup.find_all('div', attrs={'class':'question_body'})
  </code></pre>
  <p>But the simple code above is a bit problematic, as I do not want to web scrape anything that is a 'group questions'. The contents of 'group questions' (a set of different questions that are based on the same question texts) are also nested between the tags <code>&lt;div class=&quot;question_body&quot;&gt;</code> and <code>&lt;/div&gt;</code>, but the difference between 'group question' and 'non-group question' is that the source html code for the 'group question' are preceded by:</p>
  <pre class="lang-html prettyprint-override"><code>            &lt;p class=&quot;group_instructions&quot;&gt;
                  This question is a part of a group with common instructions.
                  &lt;a style=&quot;text-decoration:underline;&quot; href=&quot;/groups/4913/making-bread&quot;&gt;View group &amp;raquo;&lt;/a&gt;
              &lt;/p&gt;
  </code></pre>
  <p>for example, below is html source code for one of the group questions on the website:</p>
  <pre class="lang-html prettyprint-override"><code>            &lt;p class=&quot;group_instructions&quot;&gt;
                  This question is a part of a group with common instructions.
                  &lt;a style=&quot;text-decoration:underline;&quot; href=&quot;/groups/4913/making-bread&quot;&gt;View group &amp;raquo;&lt;/a&gt;
              &lt;/p&gt;
          
          &lt;div class=&quot;question_body&quot;&gt;
              
              
          &lt;a href=&quot;/questions/128621/which-is-not-an-ingredient-the-mother-put-in-the-bread&quot;&gt;Which is NOT an ingredient the mother put in the bread?&lt;/a&gt;
              &lt;ol&gt;
  
                      &lt;li class=&quot;answer correct&quot;&gt;
                          Sugar               
                      &lt;/li&gt;
  
                      &lt;li class=&quot;answer&quot;&gt;
                          Salt    
                      &lt;/li&gt;
  
                      &lt;li class=&quot;answer&quot;&gt;
                          Yeast
                      &lt;/li&gt;
  
                      &lt;li class=&quot;answer&quot;&gt;
                          Flour    
                      &lt;/li&gt;        
              &lt;/ol&gt;              
          &lt;/div&gt;
      &lt;/div&gt;
  </code></pre>
  <p>Note how <code>&lt;p class=&quot;group_instructions&quot;&gt;</code> precedes <code>&lt;div class=&quot;question_body&quot;&gt;</code>.
  The non-group questions are not preceded by the chunk that starts with <code>&lt;p class=&quot;group_instructions&quot;&gt;</code>.</p>
  <p>Is there any way that I can exclude the group questions from being web scraped? I do not need to stick to BeautifulSoup, if this is necessary.</p>
  <p>Thank you,</p>