AI Search Crawlers Explained: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot

AI search crawlers control whether OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot can discover and retrieve your web content

AI search crawlers are automated web clients that fetch pages so search and generative systems can discover, index, retrieve or use web content. For current AI search visibility, the crucial distinction is purpose: OAI-SearchBot is OpenAI’s search crawler, Claude-SearchBot is Anthropic’s search crawler, PerplexityBot is Perplexity‘s search crawler, and Googlebot builds the Google Search index that supports eligibility for Google AI Overviews and AI Mode.

They should not be confused with training crawlers or user-triggered fetchers. OpenAI separates OAI-SearchBot from GPTBot and ChatGPT-User; Anthropic separates Claude-SearchBot from ClaudeBot and Claude-User; Perplexity separates PerplexityBot from Perplexity-User. Google uses Googlebot for Search crawling, while Google-Extended is a separate control token for certain non-Search generative AI uses and does not control inclusion in Google Search.

NeuralAdX Ltd Editorial · Last reviewed 17 August 2026 · Scope: crawler identity, purpose, crawl controls and GEO implications.

TL;DR: allow the crawlers that support the AI search visibility you actually want

For most businesses pursuing Generative Engine Optimisation, the practical default is to keep search-oriented crawlers accessible while making an explicit, separate decision about model-training crawlers. Blocking the wrong user agent can remove or reduce a site’s eligibility for AI search discovery even when the content itself is excellent.

Contents: AI search crawler guide

The key distinction: crawling for search is not the same as crawling for training or fetching for a user

A crawler name tells you who is requesting the page; its documented purpose tells you why. That distinction now matters directly to GEO because several AI companies publish separate controls for search discovery, foundation-model training and user-triggered retrieval.

OpenAI explicitly says OAI-SearchBot and GPTBot settings are independent. A site can allow OAI-SearchBot for search visibility while disallowing GPTBot to signal that its content should not be used for training OpenAI’s generative foundation models. Anthropic publishes the same three-way separation in different names: Claude-SearchBot for search, ClaudeBot for potential training use, and Claude-User for user-directed retrieval. Perplexity separates PerplexityBot from Perplexity-User.

A simple crawler-purpose map

Mobile users: scroll horizontally to view the full table or diagram.

GEO implication: do not use one blanket rule for every AI-branded bot unless that is genuinely your policy objective. Search visibility and training preference are now separable controls on several major platforms.

OAI-SearchBot vs Claude-SearchBot vs PerplexityBot vs Googlebot: current comparison

Mobile users: scroll horizontally to view the full table or diagram.

This table deliberately separates crawler access from ranking, retrieval and citation. Allowing a crawler only creates the possibility of discovery. It does not guarantee indexing, retrieval, citation, recommendation or answer position.

OAI-SearchBot explained: the OpenAI crawler that matters for ChatGPT search visibility

OAI-SearchBot is OpenAI’s search crawler. OpenAI states that it is used to surface websites in ChatGPT’s search features and recommends allowing both the crawler in robots.txt and requests from its published IP ranges. OpenAI also states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although a navigational link may still appear in some circumstances.

The important control is that OAI-SearchBot is independent from GPTBot. GPTBot is the crawler OpenAI documents for content that may be used in training generative AI foundation models. A publisher can therefore allow search discovery without giving the same robots.txt permission to GPTBot.

OpenAI says robots.txt changes for search can take roughly 24 hours to be reflected. It also publishes an IP-range endpoint for OAI-SearchBot, which is useful when a CDN or web application firewall would otherwise block automated traffic.

Do not confuse OAI-SearchBot with GPTBot or ChatGPT-User

Mobile users: scroll horizontally to view the full table or diagram.

A useful short quotation from OpenAI is: OAI-SearchBot is for search. That one sentence captures the control boundary site owners should preserve when deciding their GEO crawl policy.

Claude-SearchBot explained: Anthropic separates search, training and user retrieval

Claude-SearchBot is Anthropic’s crawler for search quality. Anthropic says it navigates the web to improve the relevance and accuracy of search responses and that disabling it can prevent Anthropic from indexing content for search optimisation, potentially reducing visibility and accuracy in user search results.

Anthropic currently publishes three separate robot identities: Claude-SearchBot for search, ClaudeBot for web content that could potentially contribute to model training, and Claude-User for user-directed web access. The company states that these bots honor standard robots.txt directives and says it also supports the non-standard Crawl-delay extension.

That separation is useful for a commercial site: it makes it possible to express a preference that Claude can discover public content for search while restricting future training collection, if that is the publisher’s chosen policy.

Anthropic’s current documentation says its crawling should be transparent and minimally disruptive. It also explicitly states that it respects CAPTCHAs rather than attempting to bypass them. Those commitments are useful operational context when distinguishing a verified crawler from spoofed or abusive requests.

PerplexityBot explained: a search crawler with a separate user-request fetcher

PerplexityBot is Perplexity’s search crawler. Perplexity says the bot is designed to surface and link websites in Perplexity search results and is not used to crawl content for AI foundation-model training. It recommends allowing PerplexityBot in robots.txt and permitting requests from the company’s published IP ranges.

Perplexity also publishes Perplexity-User, a user-triggered fetcher that may visit a page when a person asks Perplexity a question. Perplexity says this fetcher is not used for automatic web crawling or foundation-model training and that, because the fetch was requested by a user, it generally ignores robots.txt rules. That is a material operational difference from PerplexityBot.

Perplexity states that crawler-setting changes may take up to 24 hours to propagate and recommends combining user-agent checks with its current published IP ranges when configuring a WAF.

This is a good example of why a robots.txt audit alone is not a complete crawler audit. A GEO technical review should also inspect WAF rules, bot-management settings, response codes and origin logs.

Googlebot explained: the Google Search crawler that also gates eligibility for AI Overviews and AI Mode

Googlebot is the crawler used by Google Search, and Google says a page must be indexed and eligible to appear in Search with a snippet before it can be shown as a supporting link in AI Overviews or AI Mode. Google explicitly says there are no additional technical requirements for those AI Search features beyond the underlying Search requirements.

Googlebot has smartphone and desktop variants. Google says the majority of crawl requests for most sites are made with Googlebot Smartphone because Google primarily indexes the mobile version of content. Both variants use the same Googlebot product token in robots.txt, so they cannot be selectively targeted there.

Google’s February 2026 Googlebot documentation also states that the crawler fetches the first 2 MB of a supported file type and the first 64 MB of a PDF before applying its cutoff. Large, script-heavy or poorly rendered pages can therefore create technical discovery problems even when robots.txt itself is permissive.

For Google’s generative Search experiences, the key control remains Googlebot and normal Search preview controls. Google says Google-Extended does not affect Google Search; it is a separate token relating to certain Gemini and non-Search generative AI uses.

Google also introduced dedicated Search Console generative AI performance reports to a subset of sites in June 2026, showing impressions, URLs, countries, devices and dates for generative AI features. This strengthens the case for treating Googlebot crawlability as part of a measurable AI-visibility workflow rather than as a one-time technical checkbox.

Which crawler should you allow for each AI search platform?

If the objective is discoverability in AI-generated search answers, the safest way to make crawler policy decisions is to map each platform to the bot that actually serves search. Training controls should then be treated separately where the provider exposes a separate identity.

Mobile users: scroll horizontally to view the full crawler-control table.

How robots.txt controls AI search crawlers, and what it cannot do

robots.txt is a crawler-instruction protocol, not an access-control or security system. RFC 9309 formalises the Robots Exclusion Protocol and explicitly states that its rules are not a form of access authorization. Cloudflare makes the same practical point: robots.txt expresses a preference, while technical enforcement requires controls such as WAF or bot-management rules.

Illustrative robots.txt policy for search visibility without selected training crawlers

The following is an illustrative pattern, not a universal recommendation. It suits a publisher that wants discovery in ChatGPT search, Claude search, Perplexity and Google Search, while expressing an opt-out preference for GPTBot and ClaudeBot training collection. Your legal, commercial and technical policy may differ.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Mobile users: scroll horizontally to view the full robots.txt example.

Do not copy this blindly. The correct file must account for your existing CMS paths, staging areas, private sections, search pages, faceted URLs, media resources and any other crawler rules already in place. A malformed or over-broad rule can reduce search visibility across far more than AI search.

robots.txt is only one layer: check WAF rules, CDN bot controls, HTTP status codes and crawler verification

A site can say Allow: / in robots.txt and still be effectively uncrawlable. CDNs, security plugins, rate limits, JavaScript challenges, CAPTCHAs, authentication gates, geographic blocks and bot-management rules can all return a 403, 429, challenge page or incomplete response before the crawler reaches useful content.

OpenAI recommends checking robots.txt, WAF/CDN controls and bot mitigation when OAI-SearchBot cannot access a site. Perplexity similarly documents WAF allowlisting using both its user agent and current IP ranges. Google warns that user-agent strings can be spoofed and recommends verifying Googlebot with reverse DNS or published Googlebot IP ranges.

Crawler access checklist for a GEO technical assessment

• Fetch /robots.txt from the public origin and confirm the intended user-agent groups are actually served.
• Check whether OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot receive a successful HTML response on priority pages.
• Review 403, 429, 5xx, redirect loops and bot-challenge responses in CDN/WAF and origin logs.
• Verify crawler identity against official IP ranges or the provider’s documented verification method rather than trusting a user-agent string alone.
• Confirm important page text exists in the crawlable/rendered response and is not dependent on a click, consent interaction or client-side event.
• Keep XML sitemaps current and internally link important pages so crawlers can discover them without relying on orphan URLs.
• Retest after security, CDN, caching, WordPress or robots.txt changes. Crawler access is a maintained state, not a permanent property.

Why crawler access matters for Generative Engine Optimisation, but does not guarantee an AI citation

For NeuralAdX Ltd, technical crawlability is one layer of the wider specialist discipline of Generative Engine Optimisation. GEO aims to improve how a business, website, author or entity becomes visible, retrieved, understood, trusted, cited and recommended inside AI-generated answers. Search terms such as AI SEO, AEO, LLMO, ChatGPT optimisation, Perplexity optimisation and Google AI Mode optimisation describe parts of the market, but the parent discipline remains Generative Engine Optimisation.

Crawler permission is near the beginning of the chain. A crawlable page can still fail to be indexed, fail candidate retrieval, lose in reranking, receive too little context allocation, be used without citation, or simply be irrelevant to the user’s prompt. That is why crawler configuration should be connected to retrieval testing, citation readiness, entity clarity, prompt coverage, trust signals, source selection and measurement.

For the full stage-by-stage model, see How AI Search Actually Works: The Retrieval-to-Citation Pipeline Explained. For the practical framework NeuralAdX Ltd applies across content, authority and machine readability, see the 11-Factor GEO Methodology.

Diagram: crawl permission is the first gate, not the final outcome

Mobile users: scroll horizontally to view the full table or diagram.

This is why a crawler log is evidence of access, not evidence of AI visibility. The latter needs prompt-level testing and longitudinal measurement.

Crawler data: why technical crawlability is now strategically important

Crawler traffic has become a substantial layer of modern web infrastructure. Cloudflare’s 2025 Year in Review reported that search-engine crawlers generated 40% of Verified Bot traffic and AI crawlers generated another 20%. The same report found that Googlebot represented 48.6% of crawler traffic to European customer sites in October 2025, while Bingbot accounted for 13.2% and GPTBot 10.8%.

Stacked bar: share of Cloudflare Verified Bot traffic in 2025

Mobile users: scroll horizontally to view the full table or diagram.

Bar chart: crawler traffic share for European Cloudflare customer sites, October 2025

Mobile users: scroll horizontally to view the full table or diagram.

These traffic shares are not ranking factors and not measures of citation probability. They are infrastructure-scale observations. A crawler making more requests does not mean its product will cite a page more often, and a page can be crawled without being surfaced in an answer.

How to measure whether crawler access is producing real AI search visibility

A useful GEO measurement stack separates three questions: Can the crawler fetch the page? Can the system retrieve the page for relevant prompts? Does the page or brand actually appear, get cited or influence the answer? Those are different measurements and should not be collapsed into one crawler metric.

Mobile users: scroll horizontally to view the full table or diagram.

This separation protects against a common analytical error: treating a crawler visit as a win. In GEO, access is an enabling condition. The measurable outcome is retrieval, source selection, citation and answer visibility across relevant prompts over time.

Common AI crawler mistakes that can suppress AI search visibility

• Blocking every AI-labelled bot. This can remove search crawlers as well as training crawlers and directly reduce discovery in ChatGPT, Claude or Perplexity.
• Allowing robots.txt but blocking at the WAF. The published policy says yes while the infrastructure still returns 403, 429 or a challenge page.
• Blocking Googlebot to control Google AI features. Google says Googlebot is the Search control; blocking it affects Search eligibility, including AI Search features.
• Using Google-Extended as if it controls AI Overviews or AI Mode. Google states that Google-Extended does not affect Google Search.
• Assuming a user-agent string proves identity. User-agent strings are easy to spoof. Verify against official IP ranges or provider verification guidance.
• Treating crawl frequency as visibility. Request volume is not a citation or ranking metric.
• Making primary content interaction-dependent. If important text only loads after a click, swipe, form input or unsupported client-side flow, crawlers may never receive the same content users see.
• Forgetting mobile parity. Google primarily indexes mobile content, so a weaker mobile DOM can become the version used for indexing and AI Search eligibility.

Industry Expert Quotes

“Cloudflare’s 2025 data shows search-engine crawlers generated 40% of Verified Bot traffic and AI crawlers another 20%, while Googlebot accounted for 48.6% of crawler traffic to European customer sites in October 2025. For NeuralAdX Ltd, that is why technical crawlability is part of Generative Engine Optimisation but not the end goal: allowing a crawler through the door creates eligibility; the commercial question is whether the page survives retrieval, source selection and citation for the prompts that matter.”

Paul Rowe, Founder, Chief Generative Engine Optimisation Officer & CEO, NeuralAdX Ltd

Cloudflare 2025 verified-bot dataNeuralAdX Ltd retrieval-to-citation pipeline

If crawler access is unclear, test technical eligibility and live AI retrieval together

A crawler configuration check is most useful when it is connected to actual answer behaviour. A practical next step is to compare technical crawlability with live commercial prompt testing so you can see whether the site is merely accessible or is actually being retrieved, mentioned and cited across major AI platforms and answer engines.

Frequently asked questions about OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot

The practical GEO rule: control crawler purpose deliberately, then measure what happens in the answer

OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are all important to AI-era discoverability, but they do not perform identical jobs. The safe conceptual model is to separate search discovery, training preference and user-triggered retrieval, then configure robots.txt and infrastructure controls to match the business’s actual policy.

For Generative Engine Optimisation, technical crawlability should then be tested against the outcomes that matter commercially: whether priority pages are retrieved for relevant prompts, whether the brand is mentioned, whether the domain is cited, how consistently it appears against competitors and whether those signals improve over time.

NeuralAdX Ltd publishes the AI Citation Benchmark, AI Answer Visibility & Share of Voice Benchmark, live Generative Engine Optimisation proof, and the 11-Factor GEO Methodology to keep those measurement layers separate and inspectable. You can also browse the NeuralAdX Ltd Generative Engine Optimisation research and blog archive for related technical and retrieval research.

For organisations that need implementation support rather than a generic AI SEO package, NeuralAdX Ltd specialises in Generative Engine Optimisation across AI retrieval testing, citation readiness, entity clarity, prompt coverage, trust signals, source selection, technical crawlability, AI citation benchmarking and AI answer visibility measurement.

Primary sources and update standard

Crawler names, user-agent strings, IP ranges and platform controls can change. This article uses current first-party platform documentation where available and should be rechecked after material crawler-policy changes. Infrastructure-scale statistics are clearly labelled as Cloudflare observations rather than platform ranking factors.