Crawlable

The site allows search engines and AI agents to index its pages.

Impact

About

The robots.txt file tells AI crawlers and search engines which parts of a website they can or cannot access. This helps website owners control what data is available to AI models while protecting private or sensitive information. It also prevents unnecessary crawling, reducing server load and improving site performance. By following robots.txt rules, AI crawlers respect website preferences and ensure ethical data collection.

Risks

Search engines cannot find your pages.

Why it's important

Allows AI bots to find and index your pages, helping your content appear in search results and reach more people.

User stories

As an AI bot, I want to be able to crawl and access all relevant content on the website so that I can accurately index and understand the site for providing useful responses to users.

Code

robots.txt — allow all crawlers:

User-agent: *
Allow: /

HTML meta robots tag:

<meta name="robots" content="index, follow">

Error

ScanGov messaging when a site fails a standard:

Site is not accessible to well-behaved agents.

Example

A minimal passing robots.txt that allows all crawlers and links to the sitemap:

User-agent: *
Allow: /

Sitemap: https://example.gov/sitemap.xml

Guidance

Indicators

Botability