The Web Robots Pages

This resource guide includes a FAQ about web Robots, from web users, web authors, and web robot implementors.

There is information on preventing web robots from accessing your website as well as a list of types of web robots and a mailing list. Articles on web robots are given. Links to related sites are provided.
http://www.robotstxt.org/wc/robots.html