Link Popularity & Page Rank

[Panky]

Basically the robots txt file tells a spider where it can not crawl and index.

Benefits:

• reduce 404 errors in logs (You can then help distinguish between real 404 errors and errors caused by a spider looking for the robots txt file and not finding it)
  
  protect content from theft
  
  bandwidth reduction
  
  protect data from theft
  
  block bad bots like email harvesters, and nosey people
  
  multilingual websites use robots txt to direct spiders to the right content

Drawback:

• Robots txt file can be easily found by http://www.yourdomain.com/robots.txt
  It's an open door to your site and becomes a map for people looking for information you might otherwise not want anyone to find.

You can get away with not using a robots txt file on your site. Some people choose to use the meta tags. The problem is, not all the engines read meta's, where as they all read and look for robots txt.

It's a good idea to place one on your site, even if it is simply there to control 404 errors. The spiders look for the file. Just make sure you have the syntax correct and it is uploaded to the right spot.