X Nations - View Single Post

Fyrflygrl · 08-27-2003, 11:03 AM

I asked on Cozy and I'll ask here, too: I can find all kinds of information about Robots.txt files and how to exclude certain files from being indexed.... but I can't find anything that will tell me this:

Is it necessary to have a robots.txt file? What happens if you don't have one? (i.e. if there is no robots.txt file, will (potentially) every page of the site be indexed? or will NONE of the site be indexed?)

Fyrflygrl

08-27-2003, 11:03 AM	#1
Fyrflygrl Fyrflygrl should edit this Kilts are sexy ;) Join Date: Jul 2003 Location: Rogersville, Tennessee Posts: 61 xBucks: 1,135	Help with SEO: Robots.txt I asked on Cozy and I'll ask here, too: I can find all kinds of information about Robots.txt files and how to exclude certain files from being indexed.... but I can't find anything that will tell me this: Is it necessary to have a robots.txt file? What happens if you don't have one? (i.e. if there is no robots.txt file, will (potentially) every page of the site be indexed? or will NONE of the site be indexed?) Fyrflygrl