Use your robots.txt file to direct search engine web crawlers on what to crawl.
Use your robots.txt file to direct search engine web crawlers on what to crawl. A robots.txt file can prevent your site from getting crawled and indexed, so check your setup with Google’s free robots.txt tester tool.