Crawlable
Site is available for indexing by well-behaved agents.
Indicators
Impact
(How ScanGov measures tasklist priorities.)
About
The robots.txt file tells AI crawlers and search engines which parts of a website they can or cannot access. This helps website owners control what data is available to AI models while protecting private or sensitive information. It also prevents unnecessary crawling, reducing server load and improving site performance. By following robots.txt rules, AI crawlers respect website preferences and ensure ethical data collection.
Why it's important
Allows AI bots to find and index your pages, helping your content appear in search results and reach more people.User stories
As an AI bot, I want to be able to crawl and access all relevant content on the website so that I can accurately index and understand the site for providing useful responses to users.
Error
(ScanGov messaging when a site fails a standard)
Site is not accessible to well-behaved agents.