core/src/main/java/com/digitalpebble/stormcrawler/protocol/RobotRulesParser.java
public BaseRobotRules parseRules(String url, byte[] content,
            String contentType, String robotName) {
        return robotParser.parseContent(url, content, contentType, robotName);
    }
public BaseRobotRules parseRules(String url, byte[] content,
            String contentType, String robotName) {
        return robotParser.parseContent(url, content, contentType, robotName);
    }
Parses the robots content using the {@link SimpleRobotRulesParser} from
crawler commons

@param url
A string containing url
@param content
Contents of the robots file in a byte array
@param contentType
The
@param robotName
A string containing value of
@return BaseRobotRules object