title: Python WebScraping - HTML from Selenium is not what elements inspect shows
body: <p>For a faculty project i wish to scrape some news webpages. Here i encountered a problem, because when i try to parse HTML code to python i get HTML that is in Source of a page, which is a lot different than Elements shown in Inspect page. I have used BeautifulSoup, requests and Selenium and got the same result.</p>
  
  <p>Does anyone have any idea, how i could scrape Elements of a page, if i cannot get HTML code of the page or how to get HTML code of the page to scrape it.</p>
  
  <pre><code>from selenium import webdriver
  
  url = 'https://www.24ur.com/novice/korona/v-revozu-znova-zagnali-proizvodnjo.html'
  
  driver = webdriver.Chrome()
  driver.get(url)
  htmlx = driver.execute_script("return document.documentElement.outerHTML")
  
  print(htmlx)
  </code></pre>
  
  <p>Thank you.</p>
  
  <p>Edit: 
  Here is an example of the difference.</p>
  
  <p>First part is what i get from scraping the HTML and can be also seen when viewing the page source of the page: </p>
  
  <pre><code>html lang="sl"&gt;
      &lt;head&gt;
          &lt;meta charset="utf-8"&gt;
          &lt;title&gt;Vsak dan prvi - 24ur.com&lt;/title&gt;
  
          &lt;base href="/"&gt;
  
          &lt;meta name="description" content="Vodilni slovenski spletni medij: aktualni dogodki iz Slovenije in tujine, športne novice ter ekskluzivni prispevki iz sveta zabave."&gt;
          &lt;meta http-equiv="x-ua-compatible" content="ie=edge"&gt;
          &lt;meta name="viewport" content="width=device-width, initial-scale=1.0"&gt;
          &lt;meta name="referrer" content="always" /&gt;
          &lt;link rel="icon" type="image/x-icon" href="/assets/favicons/favicon.ico"&gt;
          &lt;link rel="apple-touch-icon" sizes="180x180" href="/assets/favicons/apple-touch-icon.png"&gt;
          &lt;link rel="icon" type="image/png" href="/assets/favicons/favicon-16x16.png" sizes="16x16"&gt;
          &lt;link rel="icon" type="image/png" href="/assets/favicons/favicon-32x32.png" sizes="32x32"&gt;
          &lt;link rel="manifest" href="/assets/favicons/manifest.json"&gt;
          &lt;link rel="alternate" type="application/rss+xml" title="24ur.com RSS"  href="https://www.24ur.com/rss" /&gt;
          &lt;meta name="theme-color" content="#ffffff"&gt;
  
          &lt;link rel="preconnect" href="//gql.24ur.si"&gt;
          &lt;link rel="preconnect" href="//ads.24ur.si"&gt;
          &lt;link rel="preconnect" href="//images.24ur.si"&gt;
          &lt;link rel="preconnect" href="//fonts.gstatic.com"&gt;
          &lt;link rel="preconnect" href="//script.dotmetrics.net"&gt;
          &lt;link rel="preconnect" href="//www.google-analytics.com"&gt;
  
          &lt;link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,400,600,700|Exo+2:300&amp;amp;subset=latin-ext" rel="stylesheet"&gt;
          &lt;style&gt;body{margin:0;padding:0}.f *{padding:0;margin:0;-webkit-box-sizing:border-box;box-sizing:border-box}.f{width:100%;height:100%}.f-header{position:relative;z-index:10;position:relative;display:-webkit-box;display:-ms-flexbox;display:flex;-webkit-box-pack:justify;-ms-flex-pack:justify;justify-content:space-between;height:56px;color:#fff;background:#404faf;-webkit-box-shadow:0 2px 4px 0 rgba(0,0,0,.3);box-shadow:0 2px 4px 0 rgba(0,0,0,.3)}@media print,screen and (min-width:64em){.f-header{height:64px}}.f-logo{display:-webkit-inline-box;display:-ms-inline-flexbox;display:inline-flex;width:84px;padding:0 1rem}@media print,screen and (min-width:64em){.f-logo{border-right:1px solid #5d69bb}}.f-logo img{display:inline-block;max-width:100%;height:28px}.f-pre-hamburger{min-width:56px;border-right:1px solid #5d69bb}@media print,screen and (min-width:64em){.f-pre-hamburger{display:none!important}}.f-header-hamburger{width:18px;height:12px;position:relative}.f-header-hamburger span{position:absolute;top:50%;margin-top:-1px;width:100%;height:2px;background:#fff;-webkit-transition:all .15s ease;transition:all .15s ease}.f-header-hamburger span:first-child{-webkit-transform:translateY(-5px);transform:translateY(-5px)}.f-header-hamburger span:last-child{-webkit-transform:translateY(5px);transform:translateY(5px)}.f-main-menu{position:absolute;left:50%;-webkit-transform:translateX(-50%);transform:translateX(-50%);height:100%;display:-webkit-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-ms-flex-align:center;align-items:center}@media screen and (max-width:63.9375em){.f-main-menu{display:none!important}}.f-main-menu div{width:60px;height:12px;border-radius:2px;margin:0 .66666667rem;background:#6672bf}.f-main-menu div:first-child{width:80px}.f-main-menu div:nth-child(3){width:50px}.f-main-menu div:nth-child(5){width:90px}.f-right-menu&gt;div{min-width:64px;border-left:1px solid #5d69bb}.f-circle{width:26px;height:26px;border-radius:999em;background:#6672bf}.f-flex{display:-webkit-box;display:-ms-flexbox;display:flex}.f-xy-center{-webkit-box-pack:center;-ms-flex-pack:center;justify-content:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center}.f-load-wrap{position:fixed;top:0;right:0;bottom:0;left:0;background:#f6f6f6}.f-loader{width:48px;height:48px;border:8px solid rgba(64,79,175,.25);border-top-color:#404faf;border-radius:50%;position:relative;-webkit-animation:loader-rotate 1s linear infinite;animation:loader-rotate 1s linear infinite}@-webkit-keyframes loader-rotate{0%{-webkit-transform:rotate(0);transform:rotate(0)}100%{-webkit-transform:rotate(360deg);transform:rotate(360deg)}}@keyframes loader-rotate{0%{-webkit-transform:rotate(0);transform:rotate(0)}100%{-webkit-transform:rotate(360deg);transform:rotate(360deg)}}@media print,screen and (min-width:40em){.f-loader{width:54px;height:54px}}&lt;/style&gt;
      &lt;link rel="stylesheet" href="styles.d2d2b29314e2cc74b587.css"&gt;&lt;/head&gt;
      &lt;body class="default"&gt;
  
          &lt;div id="fb-root"&gt;&lt;/div&gt;
          &lt;onl-root&gt;
              &lt;div class="f"&gt; &lt;header class="f-header"&gt;&lt;div class="f-flex"&gt;&lt;div class="f-pre-hamburger f-flex f-xy-center"&gt;&lt;div class="f-header-hamburger"&gt; &lt;span&gt;&lt;/span&gt; &lt;span&gt;&lt;/span&gt; &lt;span&gt;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="f-logo f-xy-center"&gt; &lt;img src="data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB2ZXJzaW9uPSIxLjEiIHZpZXdCb3g9IjAgMCA5NyA0OCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHRpdGxlPmxvZ288L3RpdGxlPgo8ZGVzYz5DcmVhdGVkIHdpdGggU2tldGNoLjwvZGVzYz4KPGcgZmlsbD0ibm9uZSIgZmlsbC1ydWxlPSJldmVub2RkIj4KPHBhdGggZD0ibTc2LjkxNyAzNS4wMDhjMCA1LjAxODUtMi43OTE5IDcuNTI4OS03LjcxNjYgNy41Mjg5LTUuMTEzMSAwLTcuNDAzNS0yLjcyOTYtNy40MDM1LTcuNTI4OXYtMTUuMjQ2aDUuNjQ2OXYxNS40OTdjMCAxLjY5NDQgMC41NjQ1MyAzLjEzNzIgMi40NDY2IDMuMTM3MiAxLjkxNDQgMCAyLjMyMTItMS43NTc1IDIuMzIxMi0zLjQyMDN2LTE1LjIxNGg0LjcwNTV2MTUuMjQ2em0yLjg4NjctMTUuMjQ2aDguNDA2NGM0LjIzNDggMCA3LjQ5NzQgMS45NDUxIDcuNDk3NCA2LjMzNjggMCAyLjM1MTktMS41MzY3IDQuMzI5NC0zLjczMzMgNS4xNDM4bDMuNzMzMyAxMC44NTVoLTUuNTgzbC0yLjc5MTktOS42NjI0aC0xLjg4Mjh2OS42NjI0aC01LjY0NjF2LTIyLjMzNXptNS42NDYxIDkuMDk3MWgxLjA5ODNjMi4xMzI4IDAgMy41MTMzLTAuNjU5MTQgMy41MTMzLTIuNDQ3MyAwLTIuMzIwNC0xLjc4ODItMi41MDg4LTMuNjcxLTIuNTA4OGgtMC45NDA2MnY0Ljk1NjJ6bS00MS4wODYtMjQuNDAxaDE0LjU2djQzLjU0MmgtMTMuMDQydi01LjkzNzhsLTE5LjMyLTAuMDIyMDc2aC0yNi41NjF2LTkuNDg0M2wxNC4wNzQtMTAuNTdjMi41MzU3LTIuMTczIDUuNTU2Mi00LjcxMSA1LjU1NjItOC4zMzM5IDAtMi40MTY2LTEuNTA5OS00LjM0OTEtNC4wNDYzLTQuMzQ5MS0zLjg2NTggMC00Ljk1MyAzLjk4NTYtNS4xMzM2IDcuMTg2N2wtMTAuNDUtMi4wNTMxYzEuMTQ3Mi04Ljg3ODcgNy41NTEtMTQuNDM3IDE2LjQ5LTE0LjQzNyA4LjI3NjQgMCAxNS4yODEgNC40Njg5IDE1LjI4MSAxMy40MDkgMCA0LjQ2OTctMi4wNTIzIDcuNzkzLTUuMzE1NyAxMC42OTFsLTkuNjAyNSA3LjE4OTEtMC45NjY2NCAwLjYwMzE2IDEwLjY3Ni0wLjA0MDk5OSAxNy44MDItMjcuMzkzem0xMC4yMTggMTMuNTYyaDMuMjc5MnYtMy4yNzkyaC0zLjI3OTJ2My4yNzkyem0tOC42OTk3IDEzLjgzMXYtMTQuMjE0bC04Ljk2ODYgMTQuMjE0aDguOTY4NnptMjAuNzQyLTIyLjU1N2MwLTAuOTA0MzUtMC4yMjYyOC0yLjQ2ODYtMS40NTE1LTIuNDY4Ni0xLjQzMTggMC0xLjYzOTIgMi4wMTYxLTEuNjM5MiA0LjUwNDQgMCAyLjc1MDkgMC4xMzE2NyA0LjQ2NTggMS43MTQ5IDQuNDY1OCAxLjI0MzQgMCAxLjUyNjQtMS4wMTcxIDEuNjAxMy0yLjY5NDlsMi45NDA5IDAuMjY0MTNjMCAyLjk0MDEtMS43NTM1IDQuOTE4My00LjY5MzYgNC45MTgzLTQuMTA3IDAtNS4yMTk1LTMuNDg2NS01LjIxOTUtNi45NzIzIDAtMy4yMjMyIDEuMDkyOC02Ljk3MyA1LjIxOTUtNi45NzMgMy4yNDIxIDAgNC42OTM2IDIuMTg1NiA0LjY5MzYgNC45NTYyaC0zLjE2NjR6bTkuODkzNC00Ljk1NjFjMy45OTUxIDAgNS4yMjAzIDMuNTQyNSA1LjIyMDMgNi45NzMgMCAzLjQyOTgtMS4yMjUzIDYuOTcyMy01LjIyMDMgNi45NzIzLTMuOTk0MyAwLTUuMjE5NS0zLjU0MjUtNS4yMTk1LTYuOTcyMyAwLTMuNDMwNSAxLjIyNTMtNi45NzMgNS4yMTk1LTYuOTczem0wIDExLjQ1OGMxLjU2NDMgMCAxLjU2NDMtMS44NDY1IDEuNTY0My00LjQ4NDcgMC0yLjYzODkgMC00LjQ4NTUtMS41NjQzLTQuNDg1NS0xLjU2MzUgMC0xLjU2MzUgMS44NDY1LTEuNTYzNSA0LjQ4NTUgMCAyLjYzODIgMCA0LjQ4NDcgMS41NjM1IDQuNDg0N3ptNy4yMTc3LTExLjE5NGg0Ljc0ODhsMS40NTE1IDcuNTE5NGgwLjAzNzg0NmwxLjcxNDktNy41MTk0aDQuNjE2NHYxMy40MThoLTMuMzkxMXYtOS45MTI0aC0wLjAzNzg0NmwtMi4zMTg4IDkuOTEyNGgtMi4wNTMxbC0yLjEyOTYtMTAuMjUxaC0wLjAzNzg0NnYxMC4yNTFoLTIuNjAxMXYtMTMuNDE4eiIgZmlsbD0iI2ZmZiIvPgo8L2c+Cjwvc3ZnPgo=" alt="" /&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="f-main-menu"&gt;&lt;div&gt;&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="f-right-menu f-flex"&gt;&lt;div class="f-flex f-xy-center"&gt;&lt;div class="f-circle f-flex"&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="f-flex f-xy-center"&gt;&lt;div class="f-circle f-flex"&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt; &lt;/header&gt;&lt;div class="f-load-wrap f-flex f-xy-center"&gt;&lt;div class="f-loader"&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;
          &lt;/onl-root&gt;
  
          &lt;span id="sso_container"&gt;&lt;/span&gt;
  
  
          &lt;script src="/assets/js/ua-parser.min.js?v=2"&gt;&lt;/script&gt;
          &lt;script src="/assets/js/bootstrap_v2.js?v=3" id="bootstrap-script" data-appversion="378"&gt;&lt;/script&gt;
          &lt;script src="/assets/js/bootstrap_local.js?v=3"&gt;&lt;/script&gt;
      &lt;script src="runtime-es2015.9e836675c0540801d005.js" type="module"&gt;&lt;/script&gt;&lt;script src="runtime-es5.9e836675c0540801d005.js" nomodule defer&gt;&lt;/script&gt;&lt;script src="polyfills-es5.d9eb3ab6982041503ece.js" nomodule defer&gt;&lt;/script&gt;&lt;script src="polyfills-es2015.5e12d286e12c173da289.js" type="module"&gt;&lt;/script&gt;&lt;script src="scripts.40249a23249ab54f83bb.js" defer&gt;&lt;/script&gt;&lt;script src="main-es2015.391f0c4c090a80609edc.js" type="module"&gt;&lt;/script&gt;&lt;script src="main-es5.391f0c4c090a80609edc.js" nomodule defer&gt;&lt;/script&gt;&lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>This is an example from Inspect Element page, and this is what i would want to scrape,( i need "grid__main" class, but it is not in page source HTML: (there is a lot of code, so i just posted a screenshot of Inspect Element page)</p>
  
  <p><a href="https://i.stack.imgur.com/1RteE.png" rel="nofollow noreferrer">Inspection of Elements on this site</a></p>