title: PHP file_exists() for URL/robots.txt returns false
body: <p>I tryed to use file_exists(URL/robots.txt) to see if the file exists on randomly chosen websites and i get a false response;</p>
  
  <p>How do i check if the robots.txt file exists ?</p>
  
  <p>I dont want to start the download before i check.</p>
  
  <p>Using fopen() will do the trick ? because : Returns a file pointer resource on success, or FALSE on error.</p>
  
  <p>and i guess that i can put something like:</p>
  
  <pre><code>$f=@fopen($url,"r"); 
  if($f) ...
  </code></pre>
  
  <p>my code:</p>
  
  <p><a href="http://www1.macys.com/robots.txt" rel="nofollow">http://www1.macys.com/robots.txt</a>
  maybe it's not there
  <a href="http://www.intend.ro/robots.txt" rel="nofollow">http://www.intend.ro/robots.txt</a>
  maybe it's not there
  <a href="http://www.emag.ro/robots.txt" rel="nofollow">http://www.emag.ro/robots.txt</a>
  maybe it's not there
  <a href="http://www1.bloomingdales.com/robots.txt" rel="nofollow">http://www1.bloomingdales.com/robots.txt</a>
  maybe it's not there</p>
  
  <pre><code>try {
              if (file_exists($file)) 
                  {
                  echo 'exists'.PHP_EOL;
                  $curl_tool = new CurlTool();
                  $content = $curl_tool-&gt;fetchContent($file);
                  //if the file exists on local disk, delete it
                  if (file_exists(CRAWLER_FILES . 'robots_' . $website_id . '.txt'))
                      unlink(CRAWLER_FILES . 'robots_' . $website . '.txt');
                  echo CRAWLER_FILES . 'robots_' . $website_id . '.txt', $content . PHP_EOL;
                  file_put_contents(CRAWLER_FILES . 'robots_' . $website_id . '.txt', $content);
              }
              else
              {
                  echo 'maybe it\'s not there'.PHP_EOL;
              }
          } catch (Exception $e) {
              echo 'EXCEPTION ' . $e . PHP_EOL;
          }
  </code></pre>