title: How do you move to a new page when web scraping with BeautifulSoup?
body: <p>Below I have code that pulls the records off craigslist. Everything works great but I need to be able to go to the next set of records and repeat the same process but being new to programming I am stuck. From looking at the page code it looks like I should be clicking the arrow button contained in the span here until it contains no href: </p>
  
  <pre><code>&lt;a href="/search/syp?s=120" class="button next" title="next page"&gt;next &amp;gt; &lt;/a&gt; 
  </code></pre>
  
  <p>I was thinking that maybe this was a loop within a loop but I suppose this could be a try/except situation too. Does that sound right? How would you implement that?</p>
  
  <pre><code>import requests
  from urllib.request import urlopen
  import pandas as pd
  
  response = requests.get("https://nh.craigslist.org/d/computer-parts/search/syp")
  
  soup = BeautifulSoup(response.text,"lxml")
  
  listings = soup.find_all('li', class_= "result-row")
  
  base_url = 'https://nh.craigslist.org/d/computer-parts/search/'
  
  next_url = soup.find_all('a', class_= "button next")
  
  
  dates = []
  titles = []
  prices = []
  hoods = []
  
  while base_url !=
      for listing in listings:
          datar = listing.find('time', {'class': ["result-date"]}).text
          dates.append(datar)
  
          title = listing.find('a', {'class': ["result-title"]}).text
          titles.append(title)
  
          try:
              price = listing.find('span', {'class': "result-price"}).text
              prices.append(price)
          except:
              prices.append('missing')
  
          try:
              hood = listing.find('span', {'class': "result-hood"}).text
              hoods.append(hood)
          except:
              hoods.append('missing')
  
  #write the lists to a dataframe
  listings_df = pd.DataFrame({'Date': dates, 'Titles' : titles, 'Price' : prices, 'Location' : hoods})
  
   #write to a file
  listings_df.to_csv("craigslist_listings.csv")
  </code></pre>