Robots allowed

The robots.txt file allows search engines and AI agents to crawl the site.

On this page

    Impact

    ScanGov impact ratings

    About

    Robots allowed in a robots.txt file means search engines like Google are allowed to look at and list (or “crawl”) the page in search results. Website owners use robots.txt to tell search engines what they can and can't see. If robots are allowed, the site or page can show up in search engines, making it easier for people to find.

    Risks

    Your site is blocked from search results.

    Why it's important

    Permits search engines and web tools to access content, helping improve search visibility and gather relevant data.

    User stories

    As a search engine crawler, I want to read the robots.txt file so that I know which parts of the website I am allowed or disallowed to index.

    Code

    Example robots.txt code:

    # Only applies to search.gov scraping
User-agent: usasearch  
# Slow amount of requests
Crawl-delay: 2
# Specify it can read /archive/
Allow: /archive/

# Applies to all scrapers
User-agent: *
# Slow amount of requests to 1 every 10 seconds
Crawl-delay: 10
# Don't let them read /archive/
Disallow: /archive/

# Point to a sitemap file
Sitemap: /sitemap.xml

    Example robots meta code:

    <!-- This page can be indexed 
and links on it can be followed -->
<meta name="robots" content="index, follow">

    Example X-Robots-Tag:

    # This URL can be indexed 
# and links on it can be followed
X-Robots-Tag: index, follow

    Error

    ScanGov messaging when a site fails a standard:

    Robots policy blocks browsers or scrapers.

    Examples

    Example government website robots.txt files:

    Guidance

    Indicators

    On this page