title: python request login to website
body: <p>I am learning about how to login to websites with <code>python</code> and <code>requests</code> and went through a number of different posts on stackoverflow and videos on youtube on what is required and how to do it. </p>
  
  <p>I found that the below information is being sent through the <code>form</code> when I clicked submit on the browser. <em>I went to the developer tools under network and took a look at the headers</em></p>
  
  <p><a href="https://i.stack.imgur.com/HKAIZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HKAIZ.png" alt="form response when submitting on website"></a></p>
  
  <p><strong>What I can tell</strong></p>
  
  <p>What I can tell is that on the login page itself, they only ask for <code>username</code> and <code>password</code> of which the below code is the <code>html</code> extract.</p>
  
  <pre><code>&lt;div style="margin:0;padding:0;display:inline"&gt;&lt;input name="utf8" type="hidden" value="✓"&gt;&lt;input name="authenticity_token" type="hidden" value="T8NxfsxCHqUPzdvmM++VIpipimDyjsLHkg4Oz3Yuouk="&gt;&lt;/div&gt;
  &lt;ul class="sic_loginFailed"&gt;
      &lt;li&gt;
        &lt;label for="sic_login_header_username"&gt;Username&lt;/label&gt;
        &lt;input id="sic_login_header_username" name="name" type="text" class="sic_formText"&gt;
      &lt;/li&gt;
      &lt;li&gt;
        &lt;label for="sic_login_header_password"&gt;Password&lt;/label&gt;
        &lt;input id="sic_login_header_password" name="password" type="password" class="sic_formText"&gt;
      &lt;/li&gt;
      &lt;li class="sic_remember"&gt;
        &lt;input id="sic_login_header_remember" name="remember" type="checkbox"&gt;
        &lt;label for="sic_login_header_remember"&gt;Remember my login.&lt;/label&gt;
      &lt;/li&gt;
      &lt;li&gt;
          &lt;input type="hidden" name="redirect" 
  value="https://www.shareinvestor.com/sg"&gt;
  
  
  
        &lt;input id="sic_login_submit" type="submit" value="Sign In" class="sic_greenInputButton"&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  </code></pre>
  
  <p>So that means that <code>authenticity token</code> and <code>password_m</code> is auto generated by the website? Note: I have a feeling <code>password_m</code> is auto assigned to me when I created my account. But the token is auto generated at every login. </p>
  
  <p><strong>My question</strong></p>
  
  <p>I wrote the below code based on what I know and what I have researched, but I am still unable to log into the website. </p>
  
  <pre><code>url = "https://www.shareinvestor.com/user/login.html"  # This is the main URL login page
  
  login_data = {'name': 'test_user',
                'password': 'test_password',
                'password_m': '5d93ceb70e2bf5daa84ec3d0cd2c731a',
                'remember': True,
                'redirect': 'https://www.shareinvestor.com/sg'}
  
  with requests.Session() as s:
      a = s.get(url).text
      b = bs4.BeautifulSoup(a, 'lxml')
      c = b.findAll('input', type='hidden')  # This is to draw out the token. I tried searching for it in the cookies previously, but failed badly.... 
      for i in c:
          login_data[i['name']] = i['value']
  
      # I use the this url for the response because as per the `Headers` in the picture above, it says that this is the request URL that the form is submitting to. 
      response = requests.post('https://www.shareinvestor.com/user/do_login.html?use_https=1', data=login_data)
      response = requests.get('https://www.shareinvestor.com/user/edit_profile.html', cookies=response.cookies)
  
      print(response.text)
  </code></pre>
  
  <p>If you have read this far, I would really appreciate if you could shed some light on what I am doing right or wrong in trying to login to the website, and persist the login. </p>