If its not causing a problem let it spider. The only time we ever had a problem was years ago when we had about 200 domains with about 10,000 pages each in a single P2 400 mhz. One day T-Rex from Lycos started hitting the site so hard the machine froze up (swapping memory for run away apache processes). Ah, the good old days. 
__________________
<!--<font size="1">"Did you ever hear anyone say 'that work had better be banned because I might read it and it might be very damaging to me'?"
Joseph Henry Jackson,
American Journalist, 1894-1946</font>-->
<font size="1">"I know you believe you understand what you think I said, but I am not sure you realize that what you heard is not what I meant."<br>--Richard Nixon</font>
|
Threaded Mode