Take the 2-minute tour ×

I Have Been Writing A Robots.txt Downloader Of A List Of Website Using Python and Urllib2 . The Following is the code

    import MySQLdb
    import urllib
    import urllib2
    db = MySQLdb.connect("","root","","research" )
    cursor = db.cursor()
    sql = "SELECT * FROM sites"
         # Fetch all the rows in a list of lists.
     results = cursor.fetchall()
     for row in results:
     id = row[0]
     website = row[1]
     print website
        check = urllib2.urlopen(website,timeout=10).code
        if not check: 
            print "No WEBSERVER FOUND"
     except IOError:
        print "No Webserver Found"
     if(check==200 or clone==0):
        sql2 = "UPDATE sites SET robots_txt_available=1 WHERE ID=%s" % \
        print website," Has Robots.txt.";
    else:print website," does not Have robots.txt."
            print "Error: unable to fecth data"

            # disconnect from server

The Output of the code is :

 No Webserver Found
 Error: unable to fecth data

So it is Not Completely Executing . Can Anyone Please Tell What is the Problem in this Code.

share|improve this question
Somehow, I was expecting MixedCase variable names... –  Jon Clements Dec 16 '12 at 11:27
add comment

closed as too localized by Frédéric Hamidi, bmargulies, Wh1T3h4Ck5, Brooks Moses, Gonzalo.- Dec 16 '12 at 23:37


1 Answer

up vote 1 down vote accepted

What is your point? The given URL just does not exist and for that reason the code in the except clause is being executed. And the 'code' attribute access is only executed when there is no exception...

The proper solution is

import urllib2
   urllib2.urlopen("some url")
except urllib2.HTTPError, err:
   if err.code == 404:
share|improve this answer
Yes Thats What I Am Trying To Well. If One Website Doesnt Occur then The Rest Of the Code Doesnt Execute(i.e Rest OF Websites). –  NotToBeKnown Dec 16 '12 at 11:01
Please what? Ask a coherent question ... –  Andreas Jung Dec 16 '12 at 11:03
add comment
