title: crawler4j crawls only seed URLs
body: <p>Why does the following code build upon <code>crawler4j</code> only crawl the given seed  URLs and does not start to crawl other links?</p>
  
  <pre><code>public static void main( String[] args )
  {
        String crawlStorageFolder = "F:\\crawl";
        int numberOfCrawlers = 7;
  
        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder(crawlStorageFolder);
        config.setMaxDepthOfCrawling(4);
        /*
         * Instantiate the controller for this crawl.
         */
        PageFetcher pageFetcher = new PageFetcher(config);
  
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        robotstxtConfig.setEnabled(false);
  
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = null;
          try {
              controller = new CrawlController(config, pageFetcher, robotstxtServer);
          } catch (Exception e) {
              // TODO Auto-generated catch block
              e.printStackTrace();
          }
  
        /*
         * For each crawl, you need to add some seed urls. These are the first
         * URLs that are fetched and then the crawler starts following links
         * which are found in these pages
         */
        controller.addSeed("http://edition.cnn.com/2016/05/11/politics/paul-ryan-donald-trump-meeting/index.html");        
  
        /*
         * Start the crawl. This is a blocking operation, meaning that your code
         * will reach the line after this only when crawling is finished.
         */
        controller.start(MyCrawler.class, numberOfCrawlers);
  
    }
  </code></pre>