The Web Robots Pages

Guidelines for controlling search engine crawling.

• • This page
  • LinkedIn
  • Bluesky
  • Mastodon
  • X
  • Email

About

The Web Robots Pages provides guidelines on how websites can control search engine crawling using a robots.txt file.

Indicators

• Botability

Links

• The Web Robots Pages

Standards

• Robots valid