By 2026 a single site can log a dozen distinct AI crawlers in a week, each with its own user agent, stated purpose, and opt-out behavior. These are not Googlebot. They do not feed a traditional search index, they collect content for training and answer generation, and the question of whether to allow or block them is strategic rather than a default. The rules that worked for search crawlers, where the goal was almost always maximum visibility, do not carry over cleanly when the crawler may be building a product that competes with the content it reads.
The operators publish their crawler names and controls, so the raw information exists. What is harder is keeping the picture current: operators add crawlers, split one bot into three, and change compliance posture, and at least one has been documented crawling from undeclared user agents after being blocked. This guide maps the current landscape, the robots.txt controls, the emerging standards, and the enforcement layer for sites that need more than a request-and-hope block.
The crawlers and what they do #
The distinction that organizes everything else is training crawlers versus real-time fetch agents. Training crawlers collect content into datasets that train future models. Fetch agents retrieve a page at query time so an AI system can cite it in an answer. The opt-out decision can differ for each category, and that split is the core of a modern crawler policy.
| Crawler | Operator | Purpose | Category |
|---|---|---|---|
| GPTBot | OpenAI | Training data for future models | Training |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search citations | Search index |
| ChatGPT-User | OpenAI | Fetch when a user shares or asks about a URL | On-demand fetch |
| ClaudeBot | Anthropic | Training data for Claude models | Training |
| Claude-User | Anthropic | Fetch when a Claude user asks about a page | On-demand fetch |
| Claude-SearchBot | Anthropic | Indexing for Claude search results | Search index |
| PerplexityBot | Perplexity | Crawling for Perplexity's search index | Search index |
| Perplexity-User | Perplexity | Fetch for a specific user query | On-demand fetch |
| Google-Extended | Opt-out token for Gemini and Vertex AI training | Training (token) | |
| CCBot | Common Crawl | Open dataset used by many AI projects | Training |
| Bytespider | ByteDance | Training data for ByteDance models | Training |
| Meta-ExternalAgent | Meta | Training data for Meta AI | Training |
Each name above is a documented user agent the operator publishes, except Google-Extended, which is a robots.txt token rather than a crawler with its own IP range. OpenAI, Anthropic, Perplexity, and Common Crawl each maintain documentation naming these agents; Meta-ExternalAgent is Meta’s declared training crawler and should not be confused with the older link-preview fetcher that generates share previews. Bytespider is widely reported as a ByteDance training crawler, though ByteDance publishes little about its purpose or compliance.
The payoff of the training-versus-fetch split: a site can allow the fetch and search agents so its content can be cited in AI answers, while blocking the training crawlers so it does not feed the next generation of models. That is now the most common configuration for sites whose content is also their business.
How the opt-out works in robots.txt #
All the major AI crawlers named above state that they honor robots.txt, and each is controlled by naming its user agent explicitly.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /The User-agent: * wildcard is not a reliable way to block AI crawlers, because each bot matches its specific user agent line first and treats the wildcard as a fallback. Blocking AI crawling means enumerating each one. Anthropic also supports the non-standard Crawl-delay directive to throttle rather than block, and OpenAI publishes IP ranges so blocks can be reinforced at the firewall.
Two configurations cover most decisions. The visibility-focused pattern blocks the training crawlers while allowing the fetch and search agents:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Maximum protection enumerates every agent, training and fetch alike, each with Disallow: /. Adding a disallow rule takes effect within roughly a day for most operators. Removing content already collected into a training set is a different problem: some operators honor deletion requests, others do not, and robots.txt has no mechanism to force it.
A robots.txt block is a request, not a guarantee #
Robots.txt was written for search indexing in 1994. It carries a single binary signal, no expiration, no license terms, and no enforcement beyond a crawler choosing to comply. The gap between what site owners expect and what the protocol actually does became concrete in 2025.
In August 2025, Cloudflare reported that Perplexity continued to reach content on domains that had explicitly blocked PerplexityBot and Perplexity-User in both robots.txt and firewall rules. According to Cloudflare, the content was fetched through a generic Chrome user agent from IP addresses outside Perplexity’s documented ranges. Cloudflare responded by removing Perplexity from its verified-bot list and adding detection signatures; Perplexity disputed the characterization. Whatever the final reading, the episode is a working example of the protocol’s core limit: a declared block only stops a crawler that chooses to identify itself and obey.
That reality has pushed enforcement toward the network edge. In July 2025, Cloudflare began blocking AI crawlers by default for new sites and launched a marketplace that lets publishers charge crawlers for access rather than only allow or deny them. The default posture for a large share of the web is shifting from open to permission-based, which changes the baseline assumption behind any crawler policy written before then.
Emerging standards beyond robots.txt #
Several proposals aim to add the granularity robots.txt lacks, with uneven results.
- TDM Reservation Protocol. An HTTP header or HTML signal (
tdm-reservation: 1) reserving content from text and data mining, grounded in the European Union’s Copyright Directive. Its legal weight in the EU is stronger than its technical enforcement; crawler compliance is inconsistent. - llms.txt. A markdown file, proposed by Answer.AI’s Jeremy Howard in September 2024, that points AI systems at a site’s most useful content rather than blocking them. Adoption has stalled on the consumption side: as of 2026 no major AI provider retrieves it, and Google has stated on the record that it does not support the format.
- Meta tag directives (noai, noimageai). Page-level HTML signals meant to prevent AI training on a specific page. Major training crawlers do not consistently honor them.
- C2PA. A content-provenance framework, not an opt-out, that embeds cryptographic signatures indicating a file’s origin and edit history, addressing the inverse problem of proving provenance after AI systems alter content.
No standard has reached universal compliance. Robots.txt remains the most widely honored signal, supplemented by edge and firewall enforcement where operators do not comply, and sites with real licensing exposure tend to layer several signals so at least one is likely to be honored.
The legal landscape is moving faster than the tooling #
Robots.txt was never a legal mechanism, and the underlying question of whether AI crawlers may use web content for training without consent is being decided in court in real time. Two 2025 developments reshaped the terrain.
In Bartz v. Anthropic, Judge William Alsup ruled in June 2025 that training on legally acquired books can qualify as fair use, while training on pirated copies does not. In September 2025 Anthropic agreed to a $1.5 billion settlement over pirated training material, reported as the largest copyright settlement in United States history. The pairing is instructive: the method of acquisition, not merely the act of training, is carrying legal weight. Meanwhile, New York Times v. OpenAI and Microsoft, filed in December 2023, remains active, with the core fair-use questions for news content still unresolved.
A few implications hold regardless of how individual cases end. A site that has clearly disallowed a crawler in robots.txt sits in a stronger position than one that signaled nothing. Server logs showing crawler activity after an explicit block are potential evidence, which argues for retaining crawler logs rather than rotating them out quickly. Jurisdiction matters, since EU rules give a different footing than United States fair-use doctrine. And licensing deals are now a real branch of the landscape: OpenAI has signed content agreements with publishers including Axel Springer, News Corp, the Financial Times, and the Associated Press, with terms varying from training rights to retrieval-only access.
How permissions map to AI answer visibility #
Crawler permissions decide whether content can appear in AI-generated answers, and the mapping is specific to each system.
Google AI Overviews draw from Google’s standard search index, not from Google-Extended. Blocking Google-Extended stops content from training Gemini and Vertex AI but does not remove it from AI Overviews, because the index is shared with Search. Google has confirmed the token is not a ranking signal and does not affect Search inclusion. That makes training opt-out and answer visibility independent decisions for Google, which is the single most misunderstood point in this area.
For the others, the fetch and search agents are the levers. ChatGPT Search visibility runs through OAI-SearchBot and ChatGPT-User; Perplexity through PerplexityBot and Perplexity-User; Claude’s web answers through Claude-SearchBot and Claude-User. Blocking these excludes a site from the corresponding answer interface without touching training datasets. The underlying architecture is retrieval-augmented generation, where the model fetches relevant content at query time and synthesizes an answer from it, so the structure and clean indexability that produce good traditional SEO also produce good retrieval visibility, with crawler permission as the gate in front of both.
How to verify and monitor crawler activity #
A crawler policy is only as good as the evidence that it works, and the evidence lives in logs. Two checks matter.
Verification defends against spoofing, because a user agent string is trivial to fake. The layered test is reverse DNS plus published IP ranges: resolve the requesting IP to a hostname, forward-resolve that hostname to confirm it matches, then compare the IP against the operator’s published range where one exists. Google publishes Googlebot ranges as a JSON file; OpenAI and Anthropic have begun publishing ranges for their crawlers, though the practice is newer than for search engines. A request claiming to be GPTBot from an IP outside the published range and failing the DNS round-trip is spoofed, and rate limiting catches abusive volume regardless of declared identity.
Monitoring watches whether the configuration holds over time. The compliance test is to compare disallowed user agents against what actually requested content in the logs: a crawler that was blocked but keeps appearing is either ignoring the rule or spoofing an identity, and either finding warrants edge-level enforcement. A workable cadence is weekly review of the top crawlers by volume, monthly comparison of robots.txt rules against real log data, and a quarterly check for new crawlers and changed operator policies. The landscape drifts, and the review is what catches the drift before it compounds.
FAQ #
Does blocking Google-Extended remove my site from Google AI Overviews?
No. AI Overviews draw from the standard Google search index, and Google-Extended only controls whether content trains Gemini and Vertex AI. A site that allows Googlebot but blocks Google-Extended still appears in AI Overviews and keeps its Search ranking, since Google has stated the token is not a ranking signal.
Will the User-agent: * wildcard block all AI crawlers?
Not reliably. Each AI crawler matches its own named user agent line first and treats the wildcard as a fallback, so blocking them means listing each one explicitly. A wildcard disallow may catch a crawler that has no specific rule, but it should not be relied on for the named AI agents.
If a crawler ignores my robots.txt, what actually stops it?
Enforcement at the network edge or firewall, not robots.txt. Cloudflare and other providers block known AI crawlers at the CDN edge, and firewall rules based on published IP ranges block at the infrastructure level. The 2025 Perplexity case, where Cloudflare reported crawling from undeclared user agents, is the reason serious protection combines robots.txt with edge enforcement.
Can I allow AI answer citations while blocking AI training?
Yes, and it is the most common strategic configuration. Allow the fetch and search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) so content can be cited in answers, and disallow the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) so it does not feed model training.
AI crawler management is a policy question, not a one-time configuration. The crawlers change, the operators split and rename their bots, the legal ground shifts under active litigation, and the enforcement layer moves toward permission-based defaults. Robots.txt is the primary control and the log is the primary measurement, but the durable habit is the review itself: revisit the policy on a schedule, verify that blocks are honored, and treat the decision as one that belongs to strategy rather than to the server configuration alone.