How to Manage AI Crawlers in Your robots.txt (GPTBot, ClaudeBot, PerplexityBot)

AI companies are crawling your website with two different types of bots — retrieval bots that power AI search results, and training bots that use your content to improve their models. Knowing the difference and managing them in your robots.txt is essential for controlling how your content is used.

The Two Types of AI Crawlers

Retrieval Bots (Allow These)

These bots crawl your site in real time to answer user queries. When someone asks ChatGPT or Perplexity a question, these bots fetch relevant pages to generate answers. Blocking them means your site won’t appear in AI search results.

Bot	Used By	Purpose
ChatGPT-User	OpenAI / ChatGPT	Real-time browsing for ChatGPT responses
OAI-SearchBot	OpenAI	Powers ChatGPT search features
PerplexityBot	Perplexity AI	Fetches sources for Perplexity answers
ClaudeBot	Anthropic	Retrieval for Claude’s web access
Applebot-Extended	Apple	Powers Apple Intelligence features

Training Bots (Consider Blocking)

These bots scrape your site to add your content to training datasets. They don’t directly benefit your search visibility — they’re collecting data to improve the AI model itself.

Bot	Used By	Purpose
GPTBot	OpenAI	Training data collection
Google-Extended	Google	Training data for Gemini
CCBot	Common Crawl	Open dataset used by many AI labs
anthropic-ai	Anthropic	Training data collection
Bytespider	ByteDance	Training data for TikTok AI
FacebookBot	Meta	Training data for Meta AI

How to Configure robots.txt for AI Bots

Strategy 1: Allow Retrieval, Block Training (Recommended)

This gives you AI search visibility while opting out of training data collection.

# Allow AI search/retrieval bots
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Block AI training bots
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

Strategy 2: Allow Everything

If you want maximum AI visibility and don’t mind your content being used for training:

User-agent: *
Allow: /

This is the simplest approach and maximizes your chances of appearing in AI-generated answers. Some sites prefer this because training data inclusion can indirectly increase your brand’s presence in AI outputs.

Strategy 3: Block Everything AI

If you want to completely opt out of AI crawling:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Warning: This means your site won’t appear in AI search results from these platforms. As AI search grows, this could significantly reduce your visibility.

Do AI Crawlers Respect robots.txt?

Most major AI crawlers from established companies (OpenAI, Anthropic, Google, Perplexity) respect robots.txt directives. Smaller or less reputable bots may not.

robots.txt is a trust-based protocol — it’s a request, not an enforcement mechanism. But the major AI companies have publicly committed to respecting it, and violating it would create significant legal and PR risk for them.

How AI Crawlers Interact With Your Existing robots.txt

If your robots.txt already has a User-agent: * rule, AI bots will follow it unless you add bot-specific rules. Specific user-agent rules always take priority over the wildcard.

# This blocks everything for all bots
User-agent: *
Disallow: /admin
Disallow: /private

# But this allows PerplexityBot to crawl everything
User-agent: PerplexityBot
Allow: /

Beyond robots.txt: Other AI Access Controls

The noai and noimageai Meta Tags

Some AI companies respect meta tags that opt specific pages out of training:

<meta name="robots" content="noai, noimageai" />

These are not yet standardized, and support varies. robots.txt is still the most reliable method.

llms.txt

While robots.txt controls access, llms.txt tells AI models what your site is about and which pages to prioritize. Use both for a complete AI search strategy.

Common Mistakes

1. Blocking retrieval bots accidentally. If you add User-agent: GPTBot / Disallow: /, you’re only blocking the training bot. But some guides incorrectly conflate GPTBot with ChatGPT-User. They’re different bots with different purposes.
2. Using a blanket block and losing AI search visibility. Blocking all AI bots means Perplexity and ChatGPT won’t cite your site.
3. Not knowing which bots are visiting. Check your server logs for AI bot user agents to see who’s actually crawling your site.
4. Forgetting to update as new bots appear. AI companies regularly launch new crawlers. Review your robots.txt quarterly.

How Do I Know If AI Bots Can Access My Site?

What seoutils.dev checks: Our free audit analyzes your robots.txt specifically for AI crawler directives. We identify which retrieval bots are allowed (affecting your AI search visibility) and which training bots are blocked or allowed — so you can make informed decisions about your AI crawling policy.

Run a free audit to see your AI crawler access status.

---

This is part of our AI Search Optimization series. Related: What Is llms.txt and Why Your Website Needs One | What Is Content Citability.