Crawlable (Botability)

Site is available for indexing by well-behaved agents.

Impact

(ScanGov impact)

About

The robots.txt file tells AI crawlers and search engines which parts of a website they can or cannot access. This helps website owners control what data is available to AI models while protecting private or sensitive information. It also prevents unnecessary crawling, reducing server load and improving site performance. By following robots.txt rules, AI crawlers respect website preferences and ensure ethical data collection.

Why it's important

Allows AI bots to find and index your pages, helping your content appear in search results and reach more people.

User stories

As an AI bot, I want to be able to crawl and access all relevant content on the website so that I can accurately index and understand the site for providing useful responses to users.

Error

(ScanGov messaging when a site fails a standard)

Site is not accessible to well-behaved agents.

Guidance

• 21st Century Integrated Digital Experience Act (IDEA)
• Memorandum (M-23-22)
• The Web Robots Pages

Indicators

• Botability

• Rescan
• Share
  • This page
  • LinkedIn
  • Bluesky
  • Mastodon
  • X
  • Email