Take the 2-minute tour ×

Adding this to my robots.txt

User-agent: *
Disallow: /*action=*$

How does robots not recognizing wild cards handle this?

share|improve this question
add comment

2 Answers

Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.

This will depend a bit on the exact implementation of the crawlers robot.txt honoring scheme and can not be entirely counted on.

If you want to avoid this you could have a separate configuration for googlebot (and others who do honor robots.txt.


User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow: /*action=*$

Which bans all robots except Googlebot which will honor the wildcard configuration.

share|improve this answer
Good point with 'others who do honor robots.txt'. There are a number of SEs out there that don't play by the rules (notably Chinese and Russian spiders!!), and if you're quite serious about it, editing your .htaccess file is a more complete option. google search on blocking bad bots with htaccess –  huzzah Oct 1 '12 at 19:04
add comment

Good point Kris. (+1)

Also, I completely agreed with an htacess suggestion but I should point out that this will require some decent RegEx skills and the ability to pin point user-agent used by SE (unlike robots.txt where you just need to name the robot).

When using robots.txt you basically "transfer" all responsibility to the accessing crawler, which should identify the request and act accordingly.

On the contrary, when using htaccess, you are taking full control and manage the whole traffic on the server level. This is why you need to know all the user-agents used by a specific SE, detect a pattern and use it in your RegEx rules. (to see just how much of an issue this is, just read some forums about Baidu spider and the mess it can make)

I suggest using Botopedia to go over the relevant user-agent lists and to verify IP ranges, if needed.

share|improve this answer
add comment

Your Answer


