The Web Robots Pages

About

The Web Robots Pages provides guidelines on how websites can control search engine crawling using a robots.txt file.

• Website

Indicator

• AI-friendly
• SEO

Standards

• robots.txt