robots.txt Generator for AI Crawlers
Pick a rule for each documented AI crawler, add your own paths and sitemap, copy the robots.txt, and test any URL against any user agent to see the exact rule that wins.
Last checked
AI crawlers
- GPTBotOpenAI · TrainingCollects content that may train OpenAI models
- OAI-SearchBotOpenAI · SearchSurfaces sites in ChatGPT search
- ChatGPT-UserOpenAI · User fetchFetches a page when a ChatGPT user asksThe operator says user-requested fetches may not follow robots.txt
- ClaudeBotAnthropic · TrainingCollects content that could train Claude models
- Claude-SearchBotAnthropic · SearchIndexes pages for Claude search results
- Claude-UserAnthropic · User fetchFetches a page when a Claude user asks
- Google-ExtendedGoogle · TrainingGemini training and grounding; not Google Search
- Applebot-ExtendedApple · TrainingApple foundation model training; does not crawl itself
- PerplexityBotPerplexity · SearchIndexes pages for Perplexity answers
- Perplexity-UserPerplexity · User fetchFetches a page when a Perplexity user asksThe operator says user-requested fetches may not follow robots.txt
- Meta-ExternalAgentMeta · TrainingMeta crawler for AI training and indexing
- CCBotCommon Crawl · TrainingBuilds the Common Crawl open web archive
- Meta-ExternalFetcherMeta · User fetchFetches a link when a Meta AI user asksThe operator says user-requested fetches may not follow robots.txt
- AmazonbotAmazon · TrainingImproves Amazon services; may train Amazon AI models
- Amzn-SearchBotAmazon · SearchSearch in Amazon products such as Alexa; no training
- Amzn-UserAmazon · User fetchFetches live pages for Alexa user requestsThe operator says user-requested fetches may not follow robots.txt
- ApplebotApple · SearchSpotlight, Siri and Safari search; may also feed Apple AI
- Google-CloudVertexBotGoogle · SearchCrawls a site owner requests for Vertex AI Agents
- DuckAssistBotDuckDuckGo · SearchReal-time fetches for DuckDuckGo AI answers; no training
- MistralAI-TrainingMistral AI · TrainingBuilds datasets to train Mistral models
- MistralAI-IndexMistral AI · SearchIndexes pages for Mistral search in Vibe
- MistralAI-UserMistral AI · User fetchFetches a page when a Vibe user asks
- YouBotYou.com · SearchCrawls pages for You.com search and AI answers
Every crawler
Search engines and custom bots
No own groups yet. Search engines follow the * rules unless you add a group for them.
robots.txt
User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
Put the file at the root of the host, e.g. https://example.com/robots.txt. Allowed AI bots get the same paths as everyone else; add a custom group for different ones.
Check an existing robots.txt
- Paste a file to see which AI bots it lets in
- Typos, unsupported rules and odd paths are flagged by line
- Longest matching rule wins, as in RFC 9309
How to use it
- Choose a preset or set each crawler to Allow, Block or Same as *. Bots on Same as * get no group of their own.
- Add the paths no crawler should fetch, any Allow exceptions and your sitemap URLs. Add a group when a search engine or another bot needs different paths. The file updates as you type.
- Copy or download the file and put it at the root of your host. To audit a live file, open /robots.txt in a browser, copy it and paste it into the checker.
- Switch to Test a URL, paste a robots.txt or leave it empty to test the generated one, enter URLs or paths (one per line) and pick a user agent. The result names the group that applied and the rule line that decided it.
What it checks and why
AI companies run several crawlers with different jobs. OpenAI and Anthropic each document one bot that collects training data (GPTBot, ClaudeBot), one that indexes pages for their search features (OAI-SearchBot, Claude-SearchBot) and one that fetches a page when a user asks (ChatGPT-User, Claude-User). Each answers only to the group that names it, so blocking ClaudeBot does not block Claude-SearchBot.
Google-Extended and Applebot-Extended are not separate crawlers. They are tokens you name in robots.txt to say whether pages that Googlebot or Applebot already fetch may be used for Gemini or Apple model training. Blocking them does not change how the pages appear in Google Search or Apple search features.
The checker applies the matching rules of RFC 9309: a crawler follows the group that names its token, or the * group when none does; inside the group the longest matching path wins, and Allow wins a tie. A path with no matching rule is allowed.
Specs and sources
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Google: Google's common crawlers (Google-Extended)
- Apple: About Applebot
- Perplexity: Perplexity crawlers
- You.com: YouBot
- Meta: Meta web crawlers
- Common Crawl: CCBot
- Amazon: About Amazonbot
- Mistral AI: Crawlers
- DuckDuckGo: DuckAssistBot
- Cloudflare: Content Signals in managed robots.txt
- Google: How Google interprets robots.txt
Limits
- robots.txt is a request, not access control. The companies above say they honour it; a crawler that ignores it is not stopped. Use a login or a firewall rule for anything private.
- Blocking a training crawler stops future collection. It does not remove pages from models that were already trained.
- OpenAI, Perplexity, Meta and Amazon say their user-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) may not follow robots.txt, because a person asked for the page. Blocking them is a request at best.
- Content-Signal is a Cloudflare proposal, not part of RFC 9309. Crawlers are free to ignore it, and Google Search Console may report the line as not understood.
- The list covers crawlers whose operators document them. Other bots follow your * group. Operators rename and add bots, so check their pages from time to time.
Questions
- How do I block ChatGPT from training on my site?
- Add a group for GPTBot with Disallow: /. That is the crawler OpenAI says collects training data. OAI-SearchBot and ChatGPT-User are separate and need their own rules.
- Does blocking GPTBot or Google-Extended hurt my Google rankings?
- No. Google Search uses Googlebot, and Google says Google-Extended does not affect Search. GPTBot is OpenAI's crawler and plays no part in Google Search.
- What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?
- Anthropic says ClaudeBot collects content that could train its models, Claude-SearchBot indexes pages for search results, and Claude-User fetches a page when a Claude user asks for it.
- Can I allow AI search but block AI training?
- Yes. Use the Block training only preset: training crawlers get Disallow: / while search crawlers and user fetchers are allowed, so your pages can still be cited in AI answers.
- How do I write a robots.txt that allows all?
- Use User-agent: * followed by Allow: / (or an empty Disallow:). Both let every crawler fetch everything. The Allow all AI preset with no disallowed paths writes exactly that.
- How do I test a robots.txt file?
- Open Test a URL, paste the file, enter the URL and pick a user agent such as Googlebot or GPTBot. The tester applies RFC 9309 and shows whether the URL is allowed, which group applied and the exact rule line that won.
- Which rule wins when Allow and Disallow both match?
- The longest matching path wins, counting the characters of the rule. If an Allow and a Disallow are equally long, Allow wins. A path that matches no rule is allowed.
- Do ChatGPT-User and Perplexity-User obey robots.txt?
- Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user started the fetch, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. To keep them out, block them at your firewall or CDN as well.
- Does Google support Crawl-delay or Noindex in robots.txt?
- No. Google ignores Crawl-delay and stopped reading Noindex rules in robots.txt in 2019. Use a robots meta tag or the X-Robots-Tag header to keep a page out of the index. Bing does read Crawl-delay.
- Where does robots.txt go?
- At the root of each host, for example https://example.com/robots.txt. A subdomain needs its own file, and crawlers ignore a robots.txt placed in a subfolder.