Crawlable
Site is available for indexing by well-behaved agents.
Impact
(How ScanGov measures tasklist priorities.)
Why it's important
Crawlability ensures that search engines can index the content, making it discoverable by users.User stories
As an AI bot, I want to be able to crawl and access all relevant content on the website so that I can accurately index and understand the site for providing useful responses to users.
Error
(ScanGov messaging when a site fails a standard)
Site is not accessible to well-behaved agents.
About
The robots.txt file tells AI crawlers and search engines which parts of a website they can or cannot access. This helps website owners control what data is available to AI models while protecting private or sensitive information. It also prevents unnecessary crawling, reducing server load and improving site performance. By following robots.txt rules, AI crawlers respect website preferences and ensure ethical data collection.