scrapping website on web proxy using python
===

I am working on scrapping databases that I have access to using the duke library web proxy. I encountered the issue that since the data base is accessed through a proxy server, I can't directly scrape this database as I would if the database was did not require proxy authentication.

I tried several thing:

I wrote one script that logs into the duke network (https://shib.oit.duke.edu/idp/AuthnEngine').

I then hardcode in my login data:

    login_data = urllib.urlencode({'j_username' : 'userxx',
                               'j_password' : 'passwordxx',
                               'Submit' : 'Enter'
                               })
I then login:
    
    resp = opener.open('https://shib.oit.duke.edu/idp/AuthnEngine', login_data)

and then I create a cookie jar object to hold the cookies from proxy website.

then i try to access the database with my script and it is still telling me authentication is required. I wanted to know how I can get around the authentication required for the proxy server.

If you have any suggestions please let me know.

Thank you,
Jan