[thelist] Are search engine spiders JavaScript-enabled?

Madhu Menon webguru at vsnl.net
Fri Nov 9 13:07:34 CST 2001

At 11:02 PM 11/9/2001, Kevin wrote:

>Check your server logs.  The answer is in there.  Most legit crawlers
>clearly identify themselves.  It's also worth noting that many search sites
>(Yahoo, Google) do not use crawlers to gather site info.

That's not completely correct. Yahoo is a manual directory system and uses 
Google technology for its web site search. Google, however, *does* crawl 
sites. Its spider is called GoogleBot. How else can it index the content of 
web pages?



<<<   *   >>>
Madhu Menon
User Experience Consultant
e-mail: webguru at vsnl.net

Weblog: http://madman.weblogs.com

More information about the thelist mailing list