Well Guys!
Sweet, you are correct to your thinking that they exclude areas on your domain in question to from being spidered.
To my understanding this is all that it is used for and what I have read up on the subject of robot.txt is that the file's only fuction is to guide the spiders to where you want them to spider and to protect the dirrectories that you don't wish to have open to the public.
(when not using .htpasswd in such directory).
Panky, im calling for your imput and please correct me if i have gone astray.
|