A robots.txt file can prevent your site from getting crawled and indexed, so check your setup with Google’s free robots.txt tester tool.
Use your robots.txt file to direct search engine web crawlers on what to crawl. A robots.txt file can prevent your site from getting crawled and indexed, so check your setup with Google’s free robots.txt tester tool.