The Web Robots Pages

Guidelines for controlling search engine crawling.

Indicators

About

The Web Robots Pages provides guidelines on how websites can control search engine crawling using a robots.txt file.

Standards

Feedback