AI crawler robots.txt generator & checker
Choose which AI crawlers can read your site, including the new shared AI training opt-out, then check any robots.txt in one paste.
1. Build your robots.txt
| Crawler | Company | What it does | Your choice |
|---|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode | ||
Google-Extended | Gemini training and answers in the Gemini app. Does not affect Google Search | ||
Bingbot | Microsoft | Bing search and Copilot answers | |
OAI-SearchBot | OpenAI | Shows your pages in ChatGPT search answers | |
ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks about it | |
GPTBot | OpenAI | Collects pages to train OpenAI models | |
Claude-SearchBot | Anthropic | Claude's web search results | |
Claude-User | Anthropic | Opens a page when a Claude user asks about it | |
ClaudeBot | Anthropic | Collects pages to train Claude | |
PerplexityBot | Perplexity | Perplexity's search results | |
Perplexity-User | Perplexity | Opens a page when a Perplexity user asks about it | |
Applebot | Apple | Siri and Spotlight search | |
Applebot-Extended | Apple | Apple AI training | |
Meta-ExternalAgent | Meta | Meta AI training | |
CCBot | Common Crawl | Open web archive that many AI models train on | |
Bytespider | ByteDance | ByteDance AI and search | |
Amazonbot | Amazon | Alexa answers and Amazon AI services | |
__usage_ai-training | Shared (IETF) | One line to opt out of AI training with Google, Microsoft and OpenAI, honoured from 2027 |
Your robots.txt
Upload it to the root of your site so it opens at yourdomain.com/robots.txt. On WordPress, most SEO plugins let you edit it.
2. Check an existing robots.txt
Open yourdomain.com/robots.txt in your browser, copy everything and paste it here. Nothing is sent anywhere.
| Crawler | Result for your homepage |
|---|
Search crawlers vs training crawlers
Most AI companies now run separate crawlers. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) put your pages into AI answers with a link. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect pages to build future models. User fetchers open a page only when someone asks the assistant about it. That split lets you choose: be quoted in answers, but keep your writing out of training data.
For most businesses that want to be recommended by AI, the right answer is to allow everything. Being read is how you get named. Read more in AEO questions answered and today's news on the shared opt-out.
FAQ
How do I block AI training but stay in Google and ChatGPT search?
Block the training crawlers (Google-Extended, GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, CCBot) and add the shared __usage_ai-training token, but keep Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Choose the 'Stay in AI search, block training' preset above.
What is __usage_ai-training?
A robots.txt token from an IETF draft that tells AI companies not to train on your pages. Google, Microsoft and OpenAI said on 30 October 2026 that they will honour it, Microsoft from 1 February 2027 and Google in early 2027. A 'User-agent: *' rule does not count, so the token has to be named.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is only for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages when someone asks. Block GPTBot alone and you can still appear in ChatGPT search answers.
Can I block AI Overviews but stay in Google Search?
Not with robots.txt. AI Overviews and AI Mode use Googlebot, the same crawler as normal Search. Google-Extended only covers Gemini training and the Gemini app. Google has a separate Search Console setting for AI features.
Should my business block AI crawlers?
Usually no. If you want ChatGPT, Gemini and Perplexity to recommend you, they need to read your pages. Blocking makes sense for publishers whose writing is the product, or for private and paid content.
Want AI to say your name?
Get a free AI visibility audit. I'll test ChatGPT, Gemini, Perplexity & Google AI for your brand.