HostedSiteDemoRobots.txt syntax
User-agent: * # which crawler the rules below apply to (* = all)
Disallow: /private/ # do not crawl anything under /private/
Allow: /private/ok.html # except this one file
Sitemap: https://example.com/sitemap.xmlCrawlers obey the most specific group that names them: a User-agent: GPTBot group overrides the * group for GPTBot. Within a group, the longest matching path wins, so an Allow for a file inside a disallowed folder works. * matches any characters and $ marks the end of a URL, so Disallow: /*.pdf$ blocks all PDFs.
AI crawlers in this generator
| User-agent | Who |
|---|---|
GPTBot | OpenAI (model training) |
ClaudeBot | Anthropic (model training) |
Google-Extended | Google Gemini training (does not affect Search) |
Applebot-Extended | Apple AI training (does not affect Siri or Spotlight search) |
CCBot | Common Crawl (dataset used by many AI models) |
PerplexityBot | Perplexity (AI search index) |
Bytespider | ByteDance |
Robots.txt is a request, not a lock: reputable crawlers follow it, bad ones ignore it. For more, read the robots.txt guide.
Frequently asked questions
Does robots.txt stop a page from appearing in Google?
Not reliably. Disallow stops Google crawling a page, but if other sites link to it, the URL can still be indexed without its content. To keep a page out of search results, let it be crawled and add a noindex meta tag, or protect it with a password.
Where does robots.txt go?
At the root of the host: https://example.com/robots.txt. Each subdomain needs its own, so blog.example.com/robots.txt is a separate file. A robots.txt in a subfolder is ignored.
Will blocking AI crawlers hurt my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini training and grounding; Googlebot, which powers Search, is separate. Blocking GPTBot, ClaudeBot or CCBot does not affect any search engine ranking. Blocking PerplexityBot or similar does remove you from those AI search products.
Does Google follow Crawl-delay?
No. Googlebot ignores Crawl-delay; Bing and Yandex respect it. Google adjusts its crawl rate automatically based on how quickly your server responds.