Quote:
Originally posted by NetRodent
Even on domains where I want everything indexed, I still include a blank robots.txt. Why? Because I hate to have my error logs cluttered up with unnecessary 404 entries.
|
Think i read somewhere that you need to put something in your robots.txt file or you risk not beeing indexed at all..
As i remember this, SE robots finding an empty robots.txt file can read this as meaning you don't wan't any robot spidering at all..
not 100 % sure of this though, but might be something you wan't to check up on
