2026 AI Search Engine Crawlers: Complete Whitelist & robots.txt Guide

Key Takeaway:

Commercial AI search engines rely on dedicated crawler User-Agents distinct from classic search engine spiders. If your robots.txt contains a global Disallow: / or lacks explicit allow-rules for primary answer engines like GPTBot, ClaudeBot, and PerplexityBot, your brand is completely excluded from generative answer citations.

As answer engines like ChatGPT Search, Perplexity Sonar, and Google AI Overviews shift organic discovery from classic 10 blue links to real-time generative synthesis, your robots.txt configuration serves as the foundational gatekeeper of brand visibility.

According to empirical research by Princeton University and collaborators in Generative Engine Optimization (GEO) (Aggarwal et al., 2023), crawler indexability is the strict baseline prerequisite, and structured semantic content optimization can boost source citation visibility by up to 40% compared to unoptimized baselines.

Yet, many engineering teams inadvertently block AI retrieval agents while attempting to protect proprietary datasets from training scrapers, cutting off lucrative, high-intent referral traffic in the process.

Why Legacy robots.txt Configurations Fail in the AI Era

In classic web indexing, a simple rule like User-agent: * Allow: / was sufficient. In 2026, automated web extraction has bifurcated into two distinct operational categories:

1. Model Pre-Training Scrapers: High-volume, offline web harvesters gathering corpus data for foundational LLM training checkpoints.
2. Real-Time Answer Retrieval Crawlers: Synchronous or near-real-time user-directed agents executing on-demand queries to synthesize cited web references.

According to OpenAI's official documentation on bot management, GPTBot is used for both pre-training and search corpus expansion, whereas OAI-SearchBot is exclusively dedicated to real-time search indexing without training models on crawled content. Similarly, Anthropic's crawler guidelines distinguish interactive answer fetching (ClaudeBot and Claude-Web) from general data ingest (anthropic-ai).

If your robots.txt directive inadvertently blocks retrieval agents, LLMs cannot fetch your URLs or verify facts, forcing generative models to cite competitor alternatives.

Complete 2026 AI Crawler Registry (Tier 1 to Tier 3)

The following table categorizes the 22+ active AI crawlers and autonomous agents monitored by the OpenGEO Toolchain into three operational tiers:

Tier 1: Primary AI Answer Engines & Real-Time Retrieval

These crawlers directly feed live answer synthesis in tier-1 generative search engines:

Crawler User-Agent	Primary Operator	Function & Purpose	Recommended Directive
GPTBot	OpenAI	Training & ChatGPT corpus retrieval	Allow: /
OAI-SearchBot	OpenAI	ChatGPT Search real-time web indexing	Allow: /
ChatGPT-User	OpenAI	User-initiated on-demand web browsing sessions	Allow: /
ClaudeBot	Anthropic	Claude Search & interactive citation retrieval	Allow: /
Claude-Web	Anthropic	User-directed live URL fetching in Claude apps	Allow: /
PerplexityBot	Perplexity AI	Core web indexing & live citation grounding	Allow: /
Google-Extended	Google	Gemini & Vertex AI model training ingestion (does not affect Google Search or AI Overviews)	Policy Dependent
Applebot-Extended	Apple	Apple Intelligence search and summarization	Allow: /
DeepSeekBot	DeepSeek	DeepSeek reasoning grounding & technical search	Allow: /
Bytespider	ByteDance	Doubao & TikTok AI generative search indexing	Allow: /

Key Takeaway:

Critical Clarification: Google-Extended vs. Googlebot vs. Google AI Overviews
A prevalent misconception in technical SEO is assuming that adding User-agent: Google-Extended Disallow: / prevents your pages from appearing in Google AI Overviews. It does not. Google's documentation states that Google-Extended controls training data ingestion for Gemini models and Vertex AI APIs. Live AI Overviews in Google Search are powered by the standard Googlebot. To opt out of AI Overviews citations while maintaining Google Search indexing, use the HTML-level data-nosnippet attribute or <meta name="robots" content="nosnippet">, not robots.txt.

Tier 2: Foundation Models & Ecosystem Engines

These agents operate search, enterprise RAG, and platform-specific generative experiences:

Crawler User-Agent	Primary Operator	Function & Purpose	Recommended Directive
Amazonbot	Amazon	Alexa & Rufus AI shopping and conversational search	Allow: /
cohere-ai	Cohere	Enterprise RAG grounding and embedding data ingestion	Allow: /
Meta-ExternalAgent	Meta	Meta AI search integration and external web retrieval	Allow: /
FacebookBot	Meta	Social graph link previews and generative summaries	Allow: /
anthropic-ai	Anthropic	Foundational pre-training corpus collector	Policy Dependent
YouBot	You.com	Multi-agent live search query synthesis	Allow: /
DuckAssistBot	DuckDuckGo	DuckDuckGo generative AI instant answer cards	Allow: /

Tier 3: Open Datasets, Research & Agentic Crawlers

Crawlers powering open-source models, academic research, and custom autonomous agents:

Crawler User-Agent	Primary Operator	Function & Purpose	Recommended Directive
CCBot	Common Crawl	Open dataset foundational to open LLM training	Policy Dependent
Diffbot	Diffbot	Commercial Knowledge Graph and automated extraction	Allow: /
Baiduspider-render	Baidu	ERNIE Bot and Chinese AI search indexing	Allow: /
PetalBot	Aspiegel / Huawei	Petal Search and HarmonyOS AI Assistant indexing	Allow: /
Scrapy	Community	Universal crawler agent framework for custom AI bots	Allow: /

How to Configure an AST-Safe robots.txt Configuration

Per the formal standard IETF RFC 9309 (Robots Exclusion Protocol), crawlers match the most specific rule block for their User-Agent. If an explicit block for GPTBot is declared, the crawler ignores all directives inside User-agent: *.

Blindly appending lines like Allow: / to an existing file often creates subtle priority bugs or conflicts with existing path disallows. Depending on your current setup, choose between a Full Canonical Configuration or an Incremental AST Patch.

Option A: Full Canonical robots.txt (Recommended for Fresh Deployments)

For a clean, standards-compliant baseline that welcomes commercial AI retrieval while locking down private routes:

# ==============================================================================
# GeoFN Verified Canonical robots.txt (RFC 9309 Compliant)
# ==============================================================================

# 1. Primary AI Answer Engines (Unrestricted Public Retrieval)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: DeepSeekBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Meta-ExternalAgent
User-agent: YouBot
User-agent: DuckAssistBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /private/

# 2. General Search Spiders and Fallback Rules
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /private/

# 3. Discovery Endpoints
Sitemap: https://geofn.com/sitemap.xml
# Semantic AI Discovery (Standardized llms.txt protocol)
# See: https://geofn.com/blog/llms-txt-v2-standard-explained

Option B: Non-Destructive Incremental Patch (Preserving Existing Rules)

If your site already has an established, complex robots.txt with legacy rewrite rules, prepend explicit AI-engine blocks at the top of your file to prevent falling into restrictive fallback User-agent: * blocks:

# --- BEGIN: GeoFN AI Retrieval Whitelist (Prepend before User-agent: *) ---
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: DeepSeekBot
Allow: /
# --- END: GeoFN AI Retrieval Whitelist ---

How to Deploy in Next.js App Router

For modern React and Next.js applications, avoid maintaining static strings by utilizing the typed app/robots.ts convention:

import { MetadataRoute } from 'next';

const TIER_1_AI_AGENTS = [
  'GPTBot',
  'OAI-SearchBot',
  'ChatGPT-User',
  'ClaudeBot',
  'Claude-Web',
  'PerplexityBot',
  'Google-Extended',
  'Applebot-Extended',
  'DeepSeekBot',
  'Bytespider',
  'Amazonbot',
  'cohere-ai',
  'Meta-ExternalAgent',
  'YouBot',
  'DuckAssistBot',
];

const PRIVATE_ROUTES = ['/api/', '/admin/', '/dashboard/', '/private/'];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: TIER_1_AI_AGENTS,
        allow: '/',
        disallow: PRIVATE_ROUTES,
      },
      {
        userAgent: '*',
        allow: '/',
        disallow: PRIVATE_ROUTES,
      },
    ],
    sitemap: 'https://geofn.com/sitemap.xml',
  };
}

Frequently Asked Questions

What happens if I block GPTBot or OAI-SearchBot in robots.txt?

If you block OAI-SearchBot, your site is completely excluded from ChatGPT Search and will never be cited as an active source in ChatGPT responses. If you block GPTBot, OpenAI will not harvest your public pages for model training or secondary search corpus caching. If both are blocked, ChatGPT will rely exclusively on third-party scrapers or may hallucinate competitor information.

Should I block AI crawlers that harvest content for model training?

It depends on your business model. OpenAI, Anthropic, and Google allow granular decoupling: you can allow search and interactive answer bots (OAI-SearchBot, ClaudeBot) to capture referral traffic and citations, while disallowing pure training harvesters (GPTBot, anthropic-ai, CCBot) if proprietary IP protection is your priority.

Does blocking Google-Extended stop my site from appearing in Google AI Overviews?

No. This is one of the most common mistakes in technical SEO. Google-Extended solely controls whether Google can use your web content to train Gemini foundational models and Vertex AI commercial APIs. It has zero bearing on Google Search indexing or citation within Google AI Overviews. If Googlebot can crawl your page, your content remains eligible for AI Overviews. To restrict text snippets from appearing in AI Overviews without sacrificing Google Search ranking, apply the HTML <meta name="robots" content="max-snippet:50"> tag or the data-nosnippet attribute on specific elements.

Is robots.txt legally binding for AI retrieval agents?

No. Under IETF RFC 9309, robots.txt is an advisory protocol based on voluntary compliance. Reputable commercial operators (OpenAI, Anthropic, Google, Microsoft, Apple) strictly honor directives. However, rogue or unauthorized scrapers may ignore it. Truly sensitive data must always be protected via server-side authentication (OAuth, JWT, API keys), not robots.txt.

How does robots.txt interact with the new llms.txt standard?

robots.txt governs access permission (whether an AI crawler is allowed to visit your domain), whereas llms.txt governs consumption efficiency (providing a token-efficient Markdown map of your high-value documentation). Allowing crawlers in robots.txt is step one; publishing an llms.txt manifest is step two. Read our deep-dive: llms.txt v2 Standard Explained: Why Every SaaS Needs a Semantic Map for AI.

Inspecting Your Site in 1 Click

Verifying whether your production domain correctly unblocks these crawlers without accidental syntax regressions no longer requires manual cURL terminal tests or external regex tools.

You can inspect any website in real time using the OpenGEO Chrome Extension, which executes client-side AST parsing in your browser side panel to detect blocked AI bots, validate RFC 9309 compliance, and generate 1-click remediation patches.

---

Related Technical Guides

• What is GEO? The Complete Guide to Generative Engine Optimization (2026): The foundational engineering playbook on how AI answer engines select and cite sources.
• llms.txt v2 Standard Explained: Why Every SaaS Needs a Semantic Map for AI: After unblocking crawlers, provide an optimized semantic Markdown map of your documentation.