title: web scraping google news with python
body: <p>I am creating a web scraper for different news outlets, for Nytimes and the Guardian it was easy since they have their own API.</p>
  
  <p>Now, I want to scrape results from this newspaper GulfTimes.com. They do not provide an advanced search in their website, so I resorted to Google news. However, Google news Api has been deprecated. What i want is to retrieve the number of results from an advanced search like keyword = "Egypt" and begin_date="10/02/2011" and end_date="10/05/2011". </p>
  
  <p>This is feasible in the Google News UI just by putting the source as "Gulf Times" and the corresponding query and date and simply counting manually the number of results but when I try to do this using python, I get a 403 error which is understandable. </p>
  
  <p>Any idea on how I would do this? Or is there another service besides Google news that would allow me to do this? Keeping in mind that I would issue almost 500 requests at once. </p>
  
  <pre><code>import json
  import urllib2
  import cookielib
  import re
  from bs4 import BeautifulSoup
  
  
  def run():
     Query = "Egypt"
     Month = "3"
     FromDay = "2"
     ToDay = "4"
     Year = "13"
     url='https://www.google.com/search?pz=1&amp;cf=all&amp;ned=us&amp;hl=en&amp;tbm=nws&amp;gl=us&amp;as_q='+Query+'&amp;as_occt=any&amp;as_drrb=b&amp;as_mindate='+Month+'%2F'+FromDay+'%2F'+Year+'&amp;as_maxdate='+Month+'%2F'+ToDay+'%2F'+Year+'&amp;tbs=cdr%3A1%2Ccd_min%3A3%2F1%2F13%2Ccd_max%3A3%2F2%2F13&amp;as_nsrc=Gulf%20Times&amp;authuser=0'
     cj = cookielib.CookieJar()
     opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
     request = urllib2.Request(url)   
     response = opener.open(request)
     htmlFile = BeautifulSoup(response)
     print htmlFile
  
  
  run()
  </code></pre>