|
08-27-2003, 12:03 PM
|
#1
|
|
Fyrflygrl
should edit this
Kilts are sexy ;)
Join Date: Jul 2003
Location: Rogersville, Tennessee
Posts: 61
xBucks: 993
|
Help with SEO: Robots.txt
I asked on Cozy and I'll ask here, too: I can find all kinds of information about Robots.txt files and how to exclude certain files from being indexed.... but I can't find anything that will tell me this:
Is it necessary to have a robots.txt file? What happens if you don't have one? (i.e. if there is no robots.txt file, will (potentially) every page of the site be indexed? or will NONE of the site be indexed?)
Fyrflygrl |
|
|
|
08-27-2003, 12:59 PM
|
#2
|
|
wsjb78
should edit this
Local Shadow Agent #1
Join Date: Nov 2002
Location: Home
Posts: 1,061
xBucks: 17,216
|
if you have no robots.txt every page will be attempted to be indexed...
|
|
|
|
|
08-27-2003, 01:20 PM
|
#3
|
|
Cyndalie
is not it.
Marketing Director / SEO
Join Date: Aug 2002
Location: Philadelphia
Posts: 2,334
xBucks: 76,185
|
yep, robots.txt file is the first thing and engine will request when indexing or spidering your site, it tells it what NOT to index. Unless you want to keep spiders OUT, you dont need it.
__________________
Hardcore, Blowjobs, Gay, Wives, Lesbians, Gangbang, Movies, Tranny, Interracial, Teen....
16 Sites, $35 PPS/ 60% Revshare :: Capitalize on Experience
Cyndee - Director of Marketing & SEO |
|
|
|
|
08-27-2003, 01:32 PM
|
#4
|
|
Fyrflygrl
should edit this
Kilts are sexy ;)
Join Date: Jul 2003
Location: Rogersville, Tennessee
Posts: 61
xBucks: 993
|
Thanks MUCH wsjb78 & Cyndalie!!
I really appreciate the help 
__________________
Fyrflygrl!
Join TabooRevenue, get $100!
|
|
|
|
|
08-27-2003, 02:41 PM
|
#5
|
|
NetRodent
should edit this
X Cuseme
Join Date: Jun 2003
Location: In the walls of your house.
Posts: 154
xBucks: 383
|
Even on domains where I want everything indexed, I still include a blank robots.txt. Why? Because I hate to have my error logs cluttered up with unnecessary 404 entries.
Its also fun to use them to watch for malicious spiders. If you deny a directory in your robots.txt that nobody would have any legitimate reason to access, you can tell that any requests hitting that directory
are up to no good.
__________________
<!--<font size="1">"Did you ever hear anyone say 'that work had better be banned because I might read it and it might be very damaging to me'?"
Joseph Henry Jackson,
American Journalist, 1894-1946</font>-->
<font size="1">"I know you believe you understand what you think I said, but I am not sure you realize that what you heard is not what I meant."<br>--Richard Nixon</font>
|
|
|
|
|
08-29-2003, 09:49 AM
|
#6
|
|
Cyndalie
is not it.
Marketing Director / SEO
Join Date: Aug 2002
Location: Philadelphia
Posts: 2,334
xBucks: 76,185
|
This article goes a bit more in depth about Robots.txt and meta uses to direct spiders http://www.sitepronews.com/archives/2003/aug/29prt.html
__________________
Hardcore, Blowjobs, Gay, Wives, Lesbians, Gangbang, Movies, Tranny, Interracial, Teen....
16 Sites, $35 PPS/ 60% Revshare :: Capitalize on Experience
Cyndee - Director of Marketing & SEO |
|
|
|
|
09-01-2003, 11:10 AM
|
#7
|
|
Taass
should edit this
Senior Member
Join Date: Jun 2003
Location: Denmark
Posts: 344
xBucks: 6,679
|
Quote:
Originally posted by NetRodent
Even on domains where I want everything indexed, I still include a blank robots.txt. Why? Because I hate to have my error logs cluttered up with unnecessary 404 entries.
|
Think i read somewhere that you need to put something in your robots.txt file or you risk not beeing indexed at all..
As i remember this, SE robots finding an empty robots.txt file can read this as meaning you don't wan't any robot spidering at all..
not 100 % sure of this though, but might be something you wan't to check up on |
|
|
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is On
|
|
|
|
|
Hybrid Mode