User not logged in - login - register
Home Calendar Books School Tool Photo Gallery Message Boards Users Statistics Advertise Site Info
go to bottom | |
 Message Boards » » Robots.txt Page [1]  
Prospero
All American
11662 Posts
user info
edit post

Can you unblock Alexa so we can archive some of this site as time goes on (on archive.org)?

What's the time frame on when threads are deleted and gone forever? Was there a purge in 2003-2004?

[Edited on July 27, 2010 at 4:44 PM. Reason : .]

7/27/2010 4:42:48 PM
FroshKiller
All American
51911 Posts
user info
edit post

You know, we actually used to be way high up on Alexa before it was blocked.

7/27/2010 4:47:29 PM
lewisje
All American
9196 Posts
user info
edit post

and there's no love for Yahoo, Bing, or even Ask

7/27/2010 7:13:44 PM
adultswim
Suspended
8379 Posts
user info
edit post

Why is Google's spider blocked?

7/27/2010 8:02:25 PM
lewisje
All American
9196 Posts
user info
edit post

^It isn't: everything else is

user-agent: Mediapartners-Google*
disallow:

user-agent: Googlebot
disallow:

user-agent: *
disallow: /
This means Google an its media partners are disallowed from nowhere, while all other robots (including msnbot for Bing and Slurp for Yahoo!) are disallowed from everywhere, starting at the root of the site.

7/27/2010 8:27:09 PM
 Message Boards » Feedback Forum » Robots.txt Page [1]  
go to top | |
Admin Options : move topic | lock topic

© 2024 by The Wolf Web - All Rights Reserved.
The material located at this site is not endorsed, sponsored or provided by or on behalf of North Carolina State University.
Powered by CrazyWeb v2.39 - our disclaimer.