By doing this, you’ll pinpoint which pages the search engine crawlers should crawl and index, and prevent the indexing of pages with irrelevant content.
By doing this, you’ll pinpoint which pages the search engine crawlers should crawl and index, and prevent the indexing of pages with irrelevant content. In addition, it’s also good practice to implement canonical tags to consolidate duplicate or similar content into a single preferred version. In this way, Google will not penalise you for duplicate content.