robotstxt.org
robotstxt.org: "source for information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots. "Covers:
The Web Robots FAQ
Robots Exclusion
List of Robots
The Robots Mailing List
Articles and Papers
Related Sites
posted by Jen @ 7:28 AM
<< Home