Robots.txt?

[Blazer06]

What is this file for? It is in the root directory of alot of sites.

For example: http://www.google.com/robots.txt


Blaze

[Siliconjunkie]

robots.txt tells crawlers where they are allowed to go on your site.

It can stop them from sucking up bandwidth or preventing someone from sucking down your whole site with wget.

http://www.searchengineworld.com/rob...s_tutorial.htm

A polite crawler will obey it, others ignore it.

If you are getting alot of 404s for robots.txt it means that they are crawling your site and you don't have one.

[BeatYouWithShoe]

One thing to remember though about robots.txt that throws alot of people new to it is that you tell the bots wehre NOT to crawl, you cannot tell them where to just, just where NOT to go. Just remember that so you wont be saying later "THATS WHY MY SITE NEVER APPEARED ON GOOGLE!"

[Rand Dusing]

Yah, I had a PHPNuke website I use to run and I always wondered what it was for. But then I signed up for the Google Adsense thing and it told me to edit my robots.txt, which btw phpnuke comes adsense ready.