title: how to scrape an aspx rendered page using python
body: <p>I am scraping an aspx rendered web page <a href="http://www.tournamentsoftware.com/sport/draw.aspx?id=E880C7A5-0A60-4A98-8FF9-A3B7DD58F3E2&amp;draw=4" rel="nofollow noreferrer">Link to Page </a></p>
  
  <p>the  website is .aspx, i chose Selenium , mechanize , urllib , lxml , Beautiful soup , requests . any insights/recommendations on coding the next steps. Also used scrapy. </p>
  
  <p>I have used requests : </p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  request.get(url_to_page)
  print request.content
  </code></pre>
  
  <p>it gives </p>
  
  <pre><code>&lt;!DOCTYPE html&gt;
  &lt;html&gt;
  &lt;head&gt;&lt;meta charset="utf-8" /&gt;&lt;title&gt;
      www.tournamentsoftware.com
  &lt;/title&gt;
    &lt;style&gt;
      body {
        font: normal 12px/20px Arial, Helvetica, sans-serif;
        color: #505050;
        background: #ccc url(//static.tournamentsoftware.com/images/cw_bg.png) repeat-x;
      }
      h2 {
        font: bold 16px/16px Arial, Helvetica, sans-serif !important;
        color: #000;
        margin: 4px 0;
      }
      h4 {
        font: bold 13px/13px Arial, Helvetica, sans-serif !important;
        margin: 0 0 -8px 0;
      }
      p {
        font: normal 12px/20px Arial, Helvetica, sans-serif;
        margin: 12px 0;
      }
      p.note {
        font: normal 10px/10px Arial, Helvetica, sans-serif;
        margin: 8px 0 0 0;
        text-align: center;
        color: #999;
      }
      p.note.error {
        font: bold 13px/20px Arial, Helvetica, sans-serif;
        color: #f00;
      }
  
      .langtoggle { display:inline; margin-right:6px; }
      .langtoggle.active { display:none; }
      .langmessage { display:none; margin-bottom:20px; }
      .langmessage.active { display:block; }
      input.button {
  
        margin: 4px 0;
      }
    &lt;/style&gt;
  &lt;/head&gt;
  &lt;body&gt;
    &lt;form method="post" action="./default.aspx?returnurl=%2fsport%2fdraw.aspx%3fid%3dE880C7A5-0A60-4A98-8FF9-A3B7DD58F3E2%26draw%3d4" id="form1" class="lang1033"&gt;
  &lt;input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="p4eGoAC3005ctvGuhkv1w6Nanrs87p7iDcl4Hlk1SNw/cJovTDsJZeq54VdP4JR0injIJb59okjgeTpi30pz0LH9qjU=" /&gt;
  
  &lt;input type="hidden" name="__VIEWSTATEGENERATOR" id="__VIEWSTATEGENERATOR" value="A86F2231" /&gt;
      &lt;div id="messagewindow"&gt;
        &lt;p class="toggles"&gt;&lt;a id="Lang1033" class="lang langtoggle active" href="#" onclick="switchLang(this)"&gt;English&lt;/a&gt; &lt;/p&gt;&lt;div id="divLang1033" class="langmessage active"&gt;&lt;h2&gt;The use of cookies on www.tournamentsoftware.com&lt;/h2&gt;&lt;p&gt;We are legally obliged to get your 
          elems = document.getElementsByClassName('langmessage');
          for (var i = 0; i &lt; elems.length; i++) {
            elems[i].className = 'langmessage';
          }
          document.getElementById(AThis.id).className = 'langtoggle active';
          document.getElementById('div' + AThis.id).className = 'langmessage active';
          return false;
        }
        function toggleCookiesHelp(AElmID) {
          document.getElementById(AElmID).style.display = 'block';
          return false;
        }
        function toggleCookiesHelpByClassName() {
          var elems = document.getElementsByClassName('removecookies');
          for (var i = 0; i &lt; elems.length; i++) {
            elems[i].style.display = 'block';
          }
          elems = document.getElementsByClassName('note');
          for (var i = 0; i &lt; elems.length; i++) {
            elems[i].className = 'note error';
          }
          return false;
        }
        if (storageAvailable()) {
          if (localStorage.getItem('cookiewall')) {
            toggleCookiesHelpByClassName();
          }
          var elems = document.getElementsByClassName('button');
          for (var i = 0; i &lt; elems.length; i++) {
            elems[i].addEventListener('click', function (e) {
              localStorage.setItem('cookiewall', '1');
            });
          }
        }
        function storageAvailable() {
          try {
            var x = '__storage_test__';
            localStorage.setItem(x, x);
            localStorage.removeItem(x);
            return true;
          } catch(e) {
            return false;
          }
        }
      &lt;/script&gt;
    &lt;/form&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>Also tried with mechanize, scrapy. All of them are giving this result only. How to scrape those websites. But I am able to see the source code in the browser. Is there any way to scrape those data. </p>