Robots valid

The site's robots.txt file is written correctly so search engines can follow it.

About

Robots valid means the robots.txt file is correctly written and follows the rules that search engines understand. If it's valid, search engines can read the file and know which parts of the site they can or can't visit. A valid file helps control how your site shows up in search results and avoids errors.

This check only looks at how the file is written. It does not check whether you have one, so a site with no robots.txt is not marked invalid.

The most common error is a Sitemap line with a path instead of a full URL. Write Sitemap: https://www.example.gov/sitemap.xml, not Sitemap: /sitemap.xml. Search engines skip a Sitemap line they cannot read.

Risks

Search engines ignore crawling rules they cannot read.

Why it's important

Guides search engines on which pages to crawl or avoid, ensuring important content is indexed and irrelevant pages aren't.

Code

Example robots.txt code:

# Only applies to search.gov scraping
User-agent: usasearch  
# Slow amount of requests
Crawl-delay: 2
# Specify it can read /archive/
Allow: /archive/

# Applies to all scrapers
User-agent: *
# Slow amount of requests to 1 every 10 seconds
Crawl-delay: 10
# Don't let them read /archive/
Disallow: /archive/

# Point to a sitemap file, always with a full URL
Sitemap: https://www.example.gov/sitemap.xml

Example robots meta code:

<!-- This page can be indexed 
and links on it can be followed -->
<meta name="robots" content="index, follow">

Example X-Robots-Tag:

# This URL can be indexed 
# and links on it can be followed
X-Robots-Tag: index, follow

Error

ScanGov messaging when a site fails a standard:

Robots.txt has a syntax error.

Examples

Example government website robots.txt files:

Guidance

The Web Robots Pages

Indicators

Botability