Take the 2-minute tour ×

Update: based on Lee's comment I decided to condense my code to a really simple script and run it from the command line:

import urllib2
import sys

username = sys.argv[1]
password = sys.argv[2]
url = sys.argv[3]
print("calling %s with %s:%s\n" % (url, username, password))

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)

req = urllib2.Request(url)
f = urllib2.urlopen(req)
data = f.read()

Unfortunately it still won't generate the Authorization header (per Wireshark) :(

I'm having a problem sending basic AUTH over urllib2. I took a look at this article, and followed the example. My code:

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, "api.foursquare.com", username, password)

req = urllib2.Request("http://api.foursquare.com/v1/user")    
f = urllib2.urlopen(req)
data = f.read()

I'm seeing the following on the Wire via wireshark:

GET /v1/user HTTP/1.1
Host: api.foursquare.com
Connection: close
Accept-Encoding: gzip
User-Agent: Python-urllib/2.5 

You can see the Authorization is not sent, vs. when I send a request via curl: curl -u user:password http://api.foursquare.com/v1/user

GET /v1/user HTTP/1.1
Authorization: Basic =SNIP=
User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3
Host: api.foursquare.com
Accept: */*

For some reason my code seems to not send the authentication - anyone see what I'm missing?



share|improve this question
I wonder if the problem is that the site is not returning a 'WWW-Authenticate' header. You can check this by using try: urllib2.urlopen(req) except urllib2.HTTPError, e: print e.headers See this SO post answer. –  Mark Mikofski Mar 14 '12 at 8:41

4 Answers 4

up vote 94 down vote accepted

The problem could be that the Python libraries, per HTTP-Standard, first send an unauthenticated request, and then only if it's answered with a 401 retry, are the correct credentials sent. If the Foursquare servers don't do "totally standard authentication" then the libraries won't work.

Try using headers to do authentication:

import urllib2, base64

# You need the replace to handle encodestring adding a trailing newline 
# (https://docs.python.org/2/library/base64.html#base64.encodestring)
request.add_header("Authorization", "Basic %s" % base64string)   
result = urllib2.urlopen(request)

Had the same problem as you and found the solution from this thread: http://forums.shopify.com/categories/9/posts/27662

share|improve this answer
Thanks man, totally saved my day! –  Ulf Oct 29 '10 at 13:48
HTTP Error 505: HTTP Version Not Supported ;( –  Daniel Magnusson May 29 '12 at 12:45
Works with paypal authentication (in order to receive access_token) as well. Thanks a lot, mate! –  DerShodan May 3 at 16:00

The second parameter must be a URI, not a domain name. i.e.

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, "http://api.foursquare.com/", username, password)
share|improve this answer
Thanks - I should have mentioned I tried that in a number of different combinations http://api.foursquare.com, api.foursquare.com, http://api.foursquare.com/v1/, but that doesn't seem to solve the problem. –  Simon Mar 9 '10 at 15:02
I just tried this against a local server here that requires basic auth and with the URL in the add_password it worked fine. I would therefore suggest that something else is afoot. –  Lee Mar 9 '10 at 16:27
This will only work if the http response contains code 401 Unauthorized _and_ the header 'WWW-Authenticate'; see this SO post answer. –  Mark Mikofski Mar 14 '12 at 8:38

(copy-paste/adapted from http://stackoverflow.com/a/24048772/1733117).

First you can subclass urllib2.BaseHandler or urllib2.HTTPBasicAuthHandler, and implement http_request so that each request has the appropriate Authorization header.

import urllib2
import base64

class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
    '''Preemptive basic auth.

    Instead of waiting for a 403 to then retry with the credentials,
    send the credentials if the url is handled by the password manager.
    Note: please use realm=None when calling add_password.'''
    def http_request(self, req):
        url = req.get_full_url()
        realm = None
        # this is very similar to the code from retry_http_basic_auth()
        # but returns a request object.
        user, pw = self.passwd.find_user_password(realm, url)
        if pw:
            raw = "%s:%s" % (user, pw)
            auth = 'Basic %s' % base64.b64encode(raw).strip()
            req.add_unredirected_header(self.auth_header, auth)
        return req

    https_request = http_request

Then if you are lazy like me, install the handler globally

api_url = "http://api.foursquare.com/"
api_username = "johndoe"
api_password = "some-cryptic-value"

auth_handler = PreemptiveBasicAuthHandler()
    realm=None, # default realm.
opener = urllib2.build_opener(auth_handler)
share|improve this answer

I would suggest that the current solution is to use my package urllib2_prior_auth which solves this pretty nicely (I work on inclusion to the standard lib.

share|improve this answer

Your Answer


