title: Crawling the Google Play store
body: <p>I want to crawl the Google Play store to download the web pages of all the android application (All the webpages with the following base url: <a href="https://play.google.com/store/apps/" rel="noreferrer">https://play.google.com/store/apps/</a>). I checked the robots.txt file of the play store and it disallows crawling these URLs. </p>
  
  <p>Also, when I browse the Google Play store I can only see top applications up to 3 pages for each of the categories. How can I get the other application pages?</p>
  
  <p>If anyone has tried crawling the Google Play please let me know the following things:
  a) Were you successful in crawling the play store. If yes, please let me know how you did that.
  b) How to crawl the hidden application pages not visible in top apps for each of the categories?
  c) Is there a techniques to download the applications also and not just the webpages?</p>
  
  <p>I already searched around and found the following links:</p>
  
  <pre><code>a) https://code.google.com/p/android-market-api/ 
  b) https://code.google.com/p/android-marketplace-crawler/source/checkout 
  c) http://mohsin-junaid.blogspot.co.uk/2012/12/how-to-install-android-marketplace.html 
  d) http://mohsin-junaid.blogspot.in/2012/12/how-to-download-multiple-android-apks.html
  </code></pre>
  
  <p>Thanks!</p>