title: Access denied while scraping
body: <p>I want to create a script to go on to <a href="https://www.size.co.uk/featured/footwear/" rel="nofollow noreferrer">https://www.size.co.uk/featured/footwear/</a> and scrape the content but somehow when i run the script, i got access denied. Here is the code:</p>
  
  <pre><code>from urllib import urlopen
  from bs4 import BeautifulSoup as BS
  url = urlopen('https://www.size.co.uk/')
  print BS(url, 'lxml')
  </code></pre>
  
  <p>The output is </p>
  
  <pre><code>&lt;html&gt;&lt;head&gt;
  &lt;title&gt;Access Denied&lt;/title&gt;
  &lt;/head&gt;&lt;body&gt;
  &lt;h1&gt;Access Denied&lt;/h1&gt;
  
  You don't have permission to access "http://www.size.co.uk/" on this server.
  &lt;p&gt;
  Reference #18.6202655f.1498945327.11002828
  &lt;/p&gt;&lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>When i try it with other websites, the code works fine and also when i use Selenium, nothing happens but i still want to know how to bypass this error without using Selenium. But when i use Selenium on different website like <a href="http://www.footpatrol.co.uk/shop" rel="nofollow noreferrer">http://www.footpatrol.co.uk/shop</a> i got the same Access Denied error, here is the code for footpatrol:</p>
  
  <pre><code>from selenium import webdriver
  
  driver = webdriver.PhantomJS('C:\Users\V\Desktop\PY\web_scrape\phantomjs.exe')
  driver.get('http://www.footpatrol.com')
  pageSource = driver.page_source
  soup = BS(pageSource, 'lxml')
  print soup
  </code></pre>
  
  <p>Output is:</p>
  
  <pre><code>&lt;html&gt;&lt;head&gt;
  &lt;title&gt;Access Denied&lt;/title&gt;
  &lt;/head&gt;&lt;body&gt;
  &lt;h1&gt;Access Denied&lt;/h1&gt;
  
  You don't have permission to access "http://www.footpatrol.co.uk/" on this 
  server.&lt;p&gt;
  Reference #18.6202655f.1498945644.110590db
  
  
  &lt;/p&gt;&lt;/body&gt;&lt;/html&gt;
  </code></pre>