X Nations - View Single Post

Panky · 04-22-2004, 05:35 PM

Basically the robots txt file tells a spider where it can not crawl and index.

Benefits:

reduce 404 errors in logs (You can then help distinguish between real 404 errors and errors caused by a spider looking for the robots txt file and not finding it)

protect content from theft

bandwidth reduction

protect data from theft

block bad bots like email harvesters, and nosey people

multilingual websites use robots txt to direct spiders to the right content

Drawback:

Robots txt file can be easily found by http://www.yourdomain.com/robots.txt
It's an open door to your site and becomes a map for people looking for information you might otherwise not want anyone to find.

You can get away with not using a robots txt file on your site. Some people choose to use the meta tags. The problem is, not all the engines read meta's, where as they all read and look for robots txt.

It's a good idea to place one on your site, even if it is simply there to control 404 errors. The spiders look for the file. Just make sure you have the syntax correct and it is uploaded to the right spot.

04-22-2004, 05:35 PM	#10
Panky Panky is the Queen of Mean Moderator Join Date: Nov 2002 Location: Montréal Posts: 4,780 xBucks: 45,433	Basically the robots txt file tells a spider where it can not crawl and index. Benefits: reduce 404 errors in logs (You can then help distinguish between real 404 errors and errors caused by a spider looking for the robots txt file and not finding it) protect content from theft bandwidth reduction protect data from theft block bad bots like email harvesters, and nosey people multilingual websites use robots txt to direct spiders to the right content Drawback: Robots txt file can be easily found by http://www.yourdomain.com/robots.txt It's an open door to your site and becomes a map for people looking for information you might otherwise not want anyone to find. You can get away with not using a robots txt file on your site. Some people choose to use the meta tags. The problem is, not all the engines read meta's, where as they all read and look for robots txt. It's a good idea to place one on your site, even if it is simply there to control 404 errors. The spiders look for the file. Just make sure you have the syntax correct and it is uploaded to the right spot.