The Web Robots Pages

Guidelines for controlling search engine crawling.

Indicators

• Botability

About

The Web Robots Pages provides guidelines on how websites can control search engine crawling using a robots.txt file.

Links

• Website

Standards

• Robots valid