X Nations - View Single Post

Panky · 05-26-2004, 08:26 PM

Dr. Monaro & Sweet are correct. The robots.txt just simply tells a spider not to read a particular page or file. This isn't fool proof though. There can be errors or the spider reads a file anyway.

Be careful what you place within robots.txt . It's an open doorway for hackers to examine your site. To review any robots.txt file, simply type the domain followed by robots.txt . http://www.google.com/robots.txt

Even if you just place a blank robots.txt file in the root, it prevents 404 errors appearing in the logs for when the spider requested the file and it wasn't there. Some spiders don't bother or ignore the tag altogether.

Having one or not having one, isn't critical. Some people just use robots meta and server configurations. If you do use robots.txt, make sure it is written properly and validate it.

05-26-2004, 08:26 PM	#11
Panky Panky is the Queen of Mean Moderator Join Date: Nov 2002 Location: Montréal Posts: 4,780 xBucks: 46,666	Dr. Monaro & Sweet are correct. The robots.txt just simply tells a spider not to read a particular page or file. This isn't fool proof though. There can be errors or the spider reads a file anyway. Be careful what you place within robots.txt . It's an open doorway for hackers to examine your site. To review any robots.txt file, simply type the domain followed by robots.txt . http://www.google.com/robots.txt Even if you just place a blank robots.txt file in the root, it prevents 404 errors appearing in the logs for when the spider requested the file and it wasn't there. Some spiders don't bother or ignore the tag altogether. Having one or not having one, isn't critical. Some people just use robots meta and server configurations. If you do use robots.txt, make sure it is written properly and validate it.