How to let ChatGPT, Claude, Perplexity and Gemini crawl your website
By Cyphen Innovations. Published , last updated .
To let AI assistants find and cite your website, allow their crawlers in robots.txt: OAI-SearchBot for ChatGPT search, Claude-SearchBot and Claude-User for Claude, PerplexityBot for Perplexity, Googlebot for Google Search and AI Overviews, Google-Extended for Gemini, and bingbot for Microsoft Copilot. Pages must also be indexed to appear in AI Overviews.
Which user agents do AI search engines and assistants use?
Each AI company documents its own crawlers. The table lists the robots.txt token, what each crawler does and whether the vendor says it follows robots.txt, as published on the vendor pages in the sources below on 30 September 2026.
| Company | robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Search | Surfaces websites in ChatGPT search results | Yes |
| OpenAI | ChatGPT-User | Assistant | Fetches pages for actions a ChatGPT user starts | May not apply, per OpenAI |
| OpenAI | GPTBot | Training | Collects content to train OpenAI models | Yes |
| Anthropic | Claude-SearchBot | Search | Improves the quality of Claude search results | Yes |
| Anthropic | Claude-User | Assistant | Fetches pages when a Claude user asks a question | Yes |
| Anthropic | ClaudeBot | Training | Collects content to train Claude models | Yes |
| Perplexity | PerplexityBot | Search | Indexes pages for Perplexity answers, not for training | Yes |
| Perplexity | Perplexity-User | Assistant | Fetches pages a Perplexity user asks about | Generally ignored, per Perplexity |
| Googlebot | Search | Crawls for Google Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | Training | Token only: Gemini training and grounding, no effect on Google Search | Yes | |
| Microsoft | bingbot | Search | Crawls for Bing, which Microsoft Copilot answers draw on | Yes |
| Apple | Applebot | Search | Crawls for Spotlight, Siri and Safari search | Yes |
| Apple | Applebot-Extended | Training | Token only: opts pages out of Apple model training | Yes |
| Meta | Meta-WebIndexer | Search | Helps Meta AI cite and link to pages | Yes |
| Meta | Meta-ExternalFetcher | Assistant | Fetches pages for user requests in Meta AI | May bypass, per Meta |
| Meta | Meta-ExternalAgent | Training | Collects content for training and indexing | Yes |
| DuckDuckGo | DuckAssistBot | Assistant | Fetches pages for DuckDuckGo AI answers, not for training | Yes |
| Mistral | MistralAI-User | Assistant | Fetches pages when a Mistral user asks | Yes |
| Amazon | Amzn-SearchBot | Search | Crawls for Alexa search, not for training | Yes |
| Amazon | Amazonbot | Training | Collects content to improve Amazon products and train AI | Yes |
| Common Crawl | CCBot | Training | Builds the open Common Crawl dataset, widely used to train AI | Yes |
What is the difference between search, assistant and training crawlers?
Search crawlers build the index an AI search product answers from, such as OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. OpenAI and Perplexity both recommend allowing their search crawlers if you want to appear in their answers.
Assistant fetchers load a page when a person asks the assistant about it. OpenAI, Perplexity and Meta say these fetchers may not follow robots.txt, because a user started the request. Anthropic says its bots, including Claude-User, respect robots.txt.
Training crawlers collect content to train models, under separate tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Google says Google-Extended does not affect inclusion in Google Search, and Apple says pages disallowed for Applebot-Extended can still appear in search results.
Does blocking Google-Extended remove a site from AI Overviews?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether content is used to train Gemini models and to ground answers in Gemini Apps and on Vertex AI. AI Overviews and AI Mode are Search features: to be shown as a supporting link, a page must be indexed and eligible to appear with a snippet in Google Search.
How does Microsoft Copilot find websites?
Microsoft does not list a separate Copilot crawler. Bing's crawler, bingbot, builds the index Copilot draws on, so allow bingbot and submit your sitemap in Bing Webmaster Tools. Bing also documents two meta tags for its chat answers: NOCACHE limits them to your URL, title and snippet, and NOARCHIVE keeps a page out of them entirely.
Is there a robots.txt token for Grok?
xAI's developer documentation does not name a crawler user agent as of 30 September 2026. Under the robots.txt standard, RFC 9309, a crawler with no group of its own follows the User-agent: * group, so allowing * covers crawlers that are not named.
What does a robots.txt that allows AI search look like?
Several User-agent lines can share one rule. This example allows the search and assistant crawlers from the table, plus Google-Extended for Gemini, and keeps every other crawler allowed:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Google-Extended
User-agent: bingbot
User-agent: Applebot
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Amzn-SearchBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlTo stay in AI search but opt out of model training, add a separate group for GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot and CCBot with Disallow: /. Blocking Google-Extended also stops Gemini from grounding answers on your pages, so decide on it separately.
What else helps AI assistants cite a website?
- Get indexed by Google and Bing. Submit your sitemap in Google Search Console and Bing Webmaster Tools, because AI Overviews use pages indexed in Google Search and Copilot draws on Bing.
- Answer the question in the first paragraph under a clear heading. Assistants quote passages, and a direct answer is easier to lift than one buried in the middle of a page.
- Use structured data that matches the visible page, such as Organization, Service, FAQPage and BreadcrumbList. Google says no special schema is needed for its AI features, so treat it as help with understanding, not a switch.
- Publish an llms.txt file if you like. It is a community proposal for a Markdown index of your key pages. Google says AI Overviews do not need any AI text file, so it is optional.
- Check that firewall and bot-protection rules do not block these user agents. A robots.txt Allow cannot help if the request is refused before it reaches the page.
Sources
- OpenAI crawler documentation
- Anthropic crawler documentation
- Perplexity crawler documentation
- Google crawler documentation
- Microsoft crawler documentation
- Apple crawler documentation
- Meta crawler documentation
- DuckDuckGo crawler documentation
- Mistral crawler documentation
- Amazon crawler documentation
- Common Crawl crawler documentation
- Google: AI features and your website
- Bing: options to control content in Bing Chat (NOCACHE, NOARCHIVE)
- RFC 9309: Robots Exclusion Protocol
- The llms.txt proposal
Need help making your site visible in AI search? See our SEO services.