Robots.txt: A Practical Guide

robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may request. It controls crawling, not indexing: to keep a page out of Google you need a noindex tag instead. Used carefully it saves crawl budget and keeps crawlers out of carts and admin pages; used carelessly it can hide a whole site.

By Sumair Roudani · Published November 4, 2026

What it can and cannot do

• Can: ask reputable crawlers not to request certain paths, point them to your sitemap, and set different rules for different crawlers.
• Cannot: remove pages from search results, protect private content (the file itself is public, and lists what you are hiding), or stop crawlers that ignore it.

Syntax in one minute

User-agent: *
Disallow: /admin/
Disallow: /*?sort=
Allow: /admin/public-help/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

• User-agent starts a group of rules for a crawler; * means all.
• Disallow blocks paths that start with the value. An empty Disallow: allows everything.
• Allow makes an exception; the longest matching rule wins.
• * matches any characters and $ anchors the end of the URL.
• Sitemap can appear anywhere and takes a full URL.

Examples

WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Online shop

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /*?*filter=

Sitemap: https://example.com/sitemap.xml

Staging site

User-agent: *
Disallow: /

Better still, protect staging sites with a password, and never copy this file to production.

The mistakes that cost traffic

• Launching with the staging file: Disallow: / on a live site stops all crawling. It is the most common cause of a new site never appearing in Google.
• Using it to hide pages: a blocked page that other sites link to can still be indexed, as a bare URL with no description. Use noindex and let it be crawled.
• Blocking resources: disallowing /assets/ or /wp-includes/ can stop Google rendering your pages.
• Case and paths: rules are case-sensitive, and Disallow: /blog also blocks /blog-news/ and /blogging.

AI crawlers

AI companies use their own crawlers, which you can allow or block separately: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training, separate from Search), Applebot-Extended, CCBot (Common Crawl) and PerplexityBot. Blocking the training crawlers does not affect your search rankings. The robots.txt generator has a checkbox for each.