title: Creating a Web Crawler in Java EE
body: <p>I am creating a web crawler using Java EE Technologies. I have created a crawler service which contains the result of the WebCrawler in term <code>CrawlerElement</code> objects which contains information of interest to me. </p>
  
  <p>Currently I am using <code>JSOUP</code> Library in order to do this. But it is not reliable I am attempting the connection three times and also timeout is 10seconds still It is unreliable. </p>
  
  <p>By unreliable I mean even if it can be accessed publicly, It can not be accessed by the crawler program. I know it could be due to <code>robots.txt</code> exclusion but in that also it is allowed but still it is unrealiable. </p>
  
  <p>So I decided to go with <code>URLConnection</code> object which has <code>openConnection</code> and then <code>connect</code> method for doing this. </p>
  
  <p>I have one more requirement which is bugging me and that is : I have to get the response time in milliseconds for a CrawlerElement which means how many seconds it took to load page B from Page A?? and I checked the methods of URLConnection there is no way out in order to do that. </p>
  
  <p>Any ideas in that topic? Can anyone help me? </p>
  
  <p>I was thinking writing a code before and after which takes current time in milliseconds before the gettingContent code and current time in milliseconds subtract and save that milliseconds in database but I was thing whether it would be accurate or not? </p>
  
  <p>Thanks in advance. </p>
  
  <p>EDIT : CURRENT IMPLEMENTATION </p>
  
  <p>Current Implementation which gives me statusCode, contentType etc.. </p>
  
  <pre><code>import java.io.IOException;
  import java.net.URL;
  import java.net.URLConnection;
  
  
  public class GetContent {
  public static void main(String args[]) throws IOException {
      URL url = new URL("http://www.javacoffeebreak.com/faq/faq0079.html");
      long startTime = System.currentTimeMillis();
      URLConnection uc = url.openConnection();
      uc.setRequestProperty("Authorization", "Basic bG9hbnNkZXY6bG9AbnNkM3Y=");
      uc.setRequestProperty("User-Agent", "");
      uc.connect();
      long endTime = System.currentTimeMillis();
      System.out.println(endTime - startTime);
      String contentType = uc.getContentType();
      System.out.println(contentType);
      String statusCode = uc.getHeaderField(0);
      System.out.println(statusCode);     
     }
  }
  </code></pre>
  
  <p>what say is it okay to do this way or I should use heavy API's like Apache HttpClient or Apache Nutch.. </p>