title: NTLM authentication with Scrapy for web scraping
body: <p>I am attempting to scrape data from a website that requires authentication.<br>
  I have been able to successfully login using requests and HttpNtlmAuth with the following:</p>
  
  <pre><code>s = requests.session()     
  url = "https://website.com/things"                                                      
  response = s.get(url, auth=HttpNtlmAuth('DOMAIN\\USERNAME','PASSWORD'))
  </code></pre>
  
  <p>I would like to explore the capabilities of Scrapy, however I have not been able to successfully authenticate.  </p>
  
  <p>I came across the following middleware which seems like it could work but I do not think I have been implementing it properly:</p>
  
  <p><a href="https://github.com/reimund/ntlm-middleware/blob/master/ntlmauth.py" rel="noreferrer">https://github.com/reimund/ntlm-middleware/blob/master/ntlmauth.py</a></p>
  
  <p>In my settings.py I have </p>
  
  <pre><code>SPIDER_MIDDLEWARES = { 'test.ntlmauth.NtlmAuthMiddleware': 400, }
  </code></pre>
  
  <p>and in my spider class I have</p>
  
  <pre><code>http_user = 'DOMAIN\\USER'
  http_pass = 'PASS'
  </code></pre>
  
  <p>I have not been able to get this to work.  </p>
  
  <p>If anyone has successfully been able to scrape from a website with NTLM authentication can point me in the right direction, I would appreciate it.</p>