title: Parsing Robots.txt in python
body: <p>I want to parse robots.txt file in python. 
  I have explored robotParser and robotExclusionParser but nothing really satisfy my criteria. I want to fetch all the diallowedUrls and allowedUrls in a single shot rather then manually checking for each url if it is allowed or not. Is there any library to do this?</p>