Python Mechanize not encoding requests properly
===

Trying to automate logging into a website using Mechanize. The basic problem I'm having is that Mechanize seems to be encoding the URL to POST to improperly, causing the target server to respond in the wrong way. The URL comes from the "action" attribute of the login form.

The correct HTTP request (logged from my browser) is:

> POST /loginAction?SID=16CB42CED987B03B&WAK0Service=http/blackboard.cornell.edu@CIT.CORNELL.EDU&WAK2Name=User%20ID&WAK0Realms=NetID,WCMC-CWID,GuestID&ReturnURL=https://blackboard.cornell.edu/16CB42CED987B03B/cuwal2.c0ntinue&VerP=2&VerC=2.2.0.183/idmbuild@rhel5x64bs.cit.cornell.edu/rhel5x64bs-40/2.2.3&VerS=Apache/2.2.3%20%20Red%20Hat%20%2020120717-0511&VerO=Linux%20plean01.serverfarm.cornell.edu%202.6.18-308.1.1.el5%20%231%20SMP%20Fri%20Feb%2017%2016:51:01%20EST%202012%20x86_64%20x86_64%20x86_64%20GNU/Linux%20Red%20Hat%20Enterprise%20Linux%20Server%20release%205.8%20%20Tikanga%20&Accept=K2&WANow=1342766438&WAK2Flags=0&WAreason=1 HTTP/1.1

Here's the output of my Mechanize code:

> url argument is not a URI (contains illegal characters) 'https://web2.login.cornell.edu:443/loginAction?SID=8D90E6BEDDD4448F&WAK0Service=http/blackboard.cornell.edu@CIT.CORNELL.EDU&WAK2Name=User%20ID&WAK0Realms=NetID,WCMC-CWID,GuestID&ReturnURL=https://blackboard.cornell.edu/8D90E6BEDDD4448F/cuwal2.c0ntinue&VerP=2&VerC=2.2.0.183/idmbuild@rhel5x64bs.cit.cornell.edu/rhel5x64bs-40/2.2.3&VerS=Apache/2.2.3%20 Red%20Hat %2020120717-0511&VerO=Linux%20plean01.serverfarm.cornell.edu%202.6.18-308.1.1.el5%20%231%20SMP%20Fri%20Feb%2017%2016:51:01%20EST%202012%20x86_64%20x86_64%20x86_64%20GNU/Linux%20Red%20Hat%20Enterprise%20Linux%20Server%20release%205.8%20%20Tikanga%20&Accept=K2&WANow=1342768552&WAK2Flags=0&WAreason=1'
add_cookie_header
send: 'POST /loginAction?SID=8D90E6BEDDD4448F&WAK0Service=http/blackboard.cornell.edu@CIT.CORNELL.EDU&WAK2Name=User%20ID&WAK0Realms=NetID,WCMC-CWID,GuestID&ReturnURL=https://blackboard.cornell.edu/8D90E6BEDDD4448F/cuwal2.c0ntinue&VerP=2&VerC=2.2.0.183/idmbuild@rhel5x64bs.cit.cornell.edu/rhel5x64bs-40/2.2.3&VerS=Apache/2.2.3%20 Red%20Hat %2020120717-0511&VerO=Linux%20plean01.serverfarm.cornell.edu%202.6.18-308.1.1.el5%20%231%20SMP%20Fri%20Feb%2017%2016:51:01%20EST%202012%20x86_64%20x86_64%20x86_64%20GNU/Linux%20Red%20Hat%20Enterprise%20Linux%20Server%20release%205.8%20%20Tikanga%20&Accept=K2&WANow=1342768552&WAK2Flags=0&WAreason=1 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 54\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1\r\nHost: web2.login.cornell.edu:443\r\nReferer: https://blackboard.cornell.edu/webapps/login/?login=LOGIN\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\nnetid=user&password=password&realm=NetID&Submit=Login'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Date: Fri, 20 Jul 2012 07:15:52 GMT
header: Server: Apache/2.2.20 (Unix) mod_ssl/2.2.20 OpenSSL/0.9.8q DAV/2
header: cache-control: no-cache
header: Content-Length: 3850
header: Connection: close
header: Content-Type: text/html
extract_cookies: Date: Fri, 20 Jul 2012 07:15:52 GMT
Server: Apache/2.2.20 (Unix) mod_ssl/2.2.20 OpenSSL/0.9.8q DAV/2
cache-control: no-cache
Content-Length: 3850
Connection: close
Content-Type: text/html
Content-Type: text/html; charset=ISO-8859-1
Content-Language: en-us

The page returned says "Query string argument Accept is required (Desired credential type)" -- referring to the "Accept=K2" part of the request URL, which is being cut off because Mechanize didn't encode the URL properly and there are some spaces in there (like Apache/2.2.3%20 Red%20Hat instead of Apache/2.2.3%20%20Red%20Hat)/

How could I prevent this from happening?

My python code:

    import mechanize
    import sys, logging

    logger = logging.getLogger("mechanize")
    logger.addHandler(logging.StreamHandler(sys.stdout))
    logger.setLevel(logging.DEBUG)
    br = mechanize.Browser(factory=mechanize.RobustFactory())
    
    br.set_handle_robots(False)
    br.set_debug_http(True)
    br.set_debug_redirects(True)
    br.set_debug_responses(True)
    br.addheaders = [('user-agent', 'mozilla/5.0 (x11; ubuntu; linux x86_64; rv:13.0) gecko/20100101 firefox/13.0.1'), ('accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')]
    response = br.open("http://blackboard.cornell.edu/")
    br.select_form(name="login")
    print "clicking login"
    response1 = br.submit()
    response1.read()
    br.select_form(name="login")
    form = br.form
    form["netid"] = "username"
    form["password"] = "password"
    form.find_control(name="realm", kind="list").value = ["NetID"]
    print "entering login info and submitting"
    response2 = br.submit()
    return response2.read()


