AI Crawler Access: How to Control Which AI Bots Can Crawl, Train On, and Cite Your Site
By Mousume Akter · Founder, SEORAF
Quick answer: AI crawler access is not one single permission. Modern AI ecosystems separate training or model-development crawling, search-indexing access, and user-triggered live retrieval. A site can allow one category while restricting another. The safest governance model is therefore to identify the operator, identify the purpose of the agent, verify the operator’s current documentation, and then apply the narrowest rule that matches the actual business objective.
AI crawler governance becomes clearer when training, search indexing, and user-triggered retrieval are treated as separate decisions rather than one generic “AI bot” permission.
What “AI Crawler Access” Actually Means
AI crawler access is the set of technical and governance decisions that determine whether automated systems operated by AI companies can request, process, retrieve, index, or otherwise use content from a website. The important distinction is that these purposes are not necessarily represented by one crawler or one permission. Different operators use different agents for different jobs.
That distinction matters because a publisher may have several legitimate objectives at the same time. A site may want its content discoverable in AI-powered search while not wanting the same content collected for model-development purposes. Another site may want user-triggered retrieval because it can bring referral visibility, while having a different policy toward large-scale training collection.
The correct question is therefore not simply: “Should I block AI bots?” The better question is: “Which operator, which agent, which purpose, which content, and which business outcome am I trying to control?”
Core principle: Access policy should be purpose-specific. Do not use a broad wildcard rule when a narrower operator-specific rule can achieve the same objective.
Crawl Access vs. Indexing vs. AI Use
Crawling means requesting a resource. Indexing means processing information so it can be retrieved or surfaced by a search system. Training or model-development use is a different downstream purpose. Live retrieval is different again: an agent may request a page because a user has asked a question at that moment.
These layers can interact, but they should not be treated as identical. A crawler being allowed to fetch a page does not guarantee that the page will be indexed, cited, summarized, or used in model development. Conversely, blocking one purpose does not automatically block every other path used by the same company.
AI Crawler Fundamentals
An AI crawler is an automated HTTP client. At the network level, it can resemble other bots: it sends requests, receives responses, follows links or fetches specific URLs, and may identify itself with a user-agent string. What changes is the purpose for which the retrieved information is processed.
Training and Model-Development Crawlers
Training-oriented crawlers collect web content that may contribute to model development or related datasets. The exact downstream processing differs by operator, so publishers should use the operator’s own documentation rather than assuming that every training crawler behaves identically.
For example, Anthropic describes ClaudeBot as a crawler used to collect web content that could potentially contribute to model training. Anthropic separately documents Claude-SearchBot and Claude-User for search and user-directed retrieval.
Search-Indexing Crawlers
Search-indexing crawlers are designed to make web information available to an operator’s search or answer system. The important distinction is that search visibility and model-training access can be represented by separate controls.
Perplexity’s documentation, for example, describes PerplexityBot as a crawler designed to surface and link websites in Perplexity search results and explicitly states that it is not used to crawl content for AI foundation-model training.
Live User-Triggered Retrieval
A live retrieval agent operates differently from a conventional autonomous crawler. Instead of continuously discovering pages, it may retrieve a particular resource because a user asked a question that requires current or source-specific information.
This distinction is operationally important. A live retrieval request can happen even when a site has relatively little exposure to proactive crawling. Vendor documentation should therefore be consulted before assuming that one robots.txt rule controls every access path.
Control Tokens
Some ecosystems use a product token or control mechanism that does not represent a separate fetching crawler. Google-Extended is an important example. Google documents it as a control related to certain generative-AI training and grounding uses, while Google Search crawling remains governed by Googlebot and normal Search controls.
The Major AI and Search Crawler Ecosystem
The following directory is designed as a governance map rather than a permanent catalog. Agent names, purposes, policies, and IP ranges can change. Always treat the operator’s current documentation as the source of truth before making a consequential access decision.
| Operator | Agent / Token | Primary purpose | Governance note |
|---|---|---|---|
| OpenAI | GPTBot | Model-development / training collection | Use OpenAI’s current crawler documentation before configuring access. |
| OpenAI | OAI-SearchBot | Search discovery | Separate from GPTBot. |
| OpenAI | ChatGPT-User | User-triggered retrieval | Purpose differs from proactive training collection. |
| Anthropic | ClaudeBot | Model-development collection | Anthropic documents it separately from search and user retrieval. |
| Anthropic | Claude-SearchBot | Search | Used to improve search response relevance. |
| Anthropic | Claude-User | User-directed retrieval | Anthropic says its bots honor robots.txt. |
| Perplexity | PerplexityBot | Search | Perplexity says it is not used for foundation-model training. |
| Perplexity | Perplexity-User | User-triggered retrieval | Perplexity documents this separately and says user-requested fetching generally ignores robots.txt. |
| Googlebot | Google Search crawling | Relevant to Search and Google’s AI Search experiences. | |
| Google-Extended | AI training / grounding control | Not a separate crawling identity. | |
| Microsoft | Bingbot | Search crawling | Maintain separately from AI-specific assumptions. |
| Apple | Applebot / Applebot-Extended | Search / downstream AI-use controls | Verify current Apple documentation before implementation. |
This table should be maintained as a living governance asset. A crawler directory published once and never reviewed will eventually become inaccurate because vendors can add agents, rename products, change IP ranges, or alter the relationship between search and retrieval systems.
How AI Crawlers Discover and Access Web Content
The basic discovery process starts with a URL. Depending on the system, that URL can come from links, sitemaps, existing indexes, internal queues, search queries, or user-triggered retrieval. There is no universal AI crawler discovery mechanism.
- Discovery: the system identifies a URL that may be useful.
- Permission evaluation: a compliant crawler may retrieve and evaluate robots.txt before requesting the target resource.
- HTTP fetch: the system requests the URL.
- Response processing: status codes, headers, HTML, structured data, text and other resources may be processed according to the operator’s purpose.
- Downstream use: the information may support search, retrieval, answer generation, indexing, model development, or another documented purpose.
RFC 9309 defines robots.txt as a protocol for indicating how crawlers are requested to access URI paths. It explicitly states that robots.txt is not access authorization. A robots file should therefore never be treated as a security boundary for confidential resources.
Security rule: Never place passwords, private API endpoints, confidential files, or sensitive administrative paths behind the assumption that robots.txt will keep them private. Use authentication and application-layer access controls for actual security.
AI Search Discoverability vs. Crawl Access
One of the most important strategic distinctions is that access does not equal visibility. Allowing a crawler means that the system can potentially retrieve the content. It does not guarantee that the page will be indexed, cited, summarized, or ranked.
Google’s current documentation makes a similar distinction for AI Overviews and AI Mode. Google says there are no additional technical requirements specifically for appearing in those AI features beyond the normal Search eligibility requirements. The page must be indexed and eligible to appear in Search with a snippet.
This has an important SEO implication: AI-search optimization should not become a replacement for technical SEO. Crawlability, indexability, information architecture, helpful content, clear entities, strong internal linking and accessible page content remain foundational.
Visibility principle: Crawl access is an eligibility layer, not a ranking guarantee.
The Decision Framework Before robots.txt
Before writing a robots.txt rule, document the business decision. This prevents accidental overblocking and makes future maintenance easier.
| Question | If yes | If no |
|---|---|---|
| Do we want search visibility? | Preserve the relevant search crawler access. | Evaluate the search-specific control. |
| Do we want user-triggered retrieval? | Preserve the relevant user agent where applicable. | Restrict that user agent according to vendor rules. |
| Do we permit model-development collection? | Allow the relevant training crawler. | Disallow the relevant training crawler. |
| Is the content private? | Use normal public web controls. | Use authentication or application security; do not rely on robots.txt. |
| Is the decision reversible? | Document the policy and review date. | Use additional safeguards and obtain technical review. |
Recommended Policy Model
- Define the content’s business value.
- Define whether search visibility matters.
- Define whether user-triggered retrieval matters.
- Define whether model-development collection is acceptable.
- Check the vendor’s current documentation.
- Apply the narrowest effective rule.
- Validate server and WAF behavior.
- Monitor logs after deployment.
- Record the decision owner and review date.
robots.txt Directives and Examples
robots.txt is most useful when the publisher wants to communicate crawler-access preferences. The protocol uses user-agent groups and path-based Allow and Disallow rules. RFC 9309 specifies that the most specific matching path is used.
Block a Specific Training Crawler
User-agent: GPTBot
Disallow: /This example expresses a site-wide disallow for the named product token. Before deploying it, verify the operator’s current documentation and confirm that the user-agent token is still correct.
Allow Search but Restrict Training
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /This pattern demonstrates the principle of purpose-specific governance. Anthropic currently documents separate bots for model development, search and user-directed retrieval, and says its bots honor robots.txt.
Longest Match
User-agent: ExampleBot
Disallow: /assets/
Allow: /assets/public/For a request such as /assets/public/logo.png, the more specific matching path is /assets/public/. RFC 9309 defines this specificity rule rather than making the result dependent on the visual order of the two path rules.
Do Not Treat robots.txt as a Security Wall
RFC 9309 explicitly says that robots.txt rules are not access authorization. A blocked path can remain publicly discoverable because the robots file itself is public and because other systems may not honor the protocol. Sensitive resources should therefore be protected with authentication or application-level controls.
Crawl-delay Requires Vendor Verification
Crawl-delay is not part of the core RFC 9309 Allow/Disallow protocol. Individual operators may document support for it. Anthropic, for example, currently documents support for the non-standard Crawl-delay extension.
Do not add Crawl-delay to a global policy and assume that every crawler will honor it. Treat it as vendor-specific behavior.
Googlebot, AI Overviews, AI Mode and Google-Extended
Google’s AI Search ecosystem is frequently misunderstood because publishers sometimes assume that AI Overviews and AI Mode require a separate “AI Googlebot.” Google’s documentation instead explains that AI features in Search remain connected to the normal Search ecosystem.
Google states that its AI features use the same foundational SEO principles as Search. To be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Google Search with a snippet. There are no additional technical requirements specifically required for AI Overviews or AI Mode.
Googlebot and Search
For Search visibility, the relevant governance layer remains Google Search’s normal crawling and indexing system. If a publisher blocks Googlebot, the consequence is not limited to an AI feature; it can affect ordinary Search crawling and indexing as well.
Google-Extended Is Different
Google-Extended is a separate product token used to communicate publisher preferences for certain generative-AI training and grounding uses. Google’s documentation has also clarified that Google-Extended does not control Google Search itself.
Do not confuse these controls: Googlebot controls Search crawling. Google-Extended is a separate downstream AI-use control. Blocking one should not be described as automatically blocking the other.
AI Overviews and AI Mode Are Still SEO
For SEORAF publishers, the practical lesson is straightforward: do not replace technical SEO with a collection of speculative “GEO hacks.” Google explicitly continues to recommend foundational SEO practices for its generative AI Search experiences.
That means pages intended to be useful in AI-powered Search should still be: crawlable, indexable, understandable, useful, internally connected, technically sound, and supported by clear information architecture.
AI Overviews and AI Mode Are Still SEO
For SEORAF publishers, the practical lesson is straightforward: do not replace technical SEO with a collection of speculative “GEO hacks.” Google explicitly continues to recommend foundational SEO practices for its generative AI Search experiences.
That means pages intended to be useful in AI-powered Search should still be: crawlable, indexable, understandable, useful, internally connected, technically sound, and supported by clear information architecture.
OpenAI Crawler Governance
OpenAI uses different crawler identities for different purposes. The key governance lesson is that GPTBot should not automatically be treated as interchangeable with every other OpenAI web agent.
For production governance, maintain a dedicated OpenAI section in the site’s crawler policy and verify the current OpenAI documentation before publishing or modifying rules.
Separate the Purposes
- Training/model-development: distinguish the training crawler.
- Search: distinguish the search crawler.
- User retrieval: distinguish the user-triggered fetcher.
- Other product functions: verify whether additional agents have been introduced.
This separation prevents a common mistake: blocking a training crawler and then assuming that the site has also blocked every path through which an AI search product could discover or retrieve the site’s content.
Anthropic Crawler Governance
Anthropic currently documents three distinct robots: ClaudeBot, Claude-SearchBot and Claude-User. Their documented purposes are different, which makes purpose-specific robots.txt governance possible.
| Agent | Purpose | Effect of restriction |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training. | Anthropic says restricting it signals that future material should be excluded from model-training datasets. |
| Claude-SearchBot | Improves search result quality. | Restriction can reduce search visibility and accuracy. |
| Claude-User | User-directed web retrieval. | Restriction prevents retrieval for those user-directed requests. |
Anthropic also states that its bots honor robots.txt and documents support for the non-standard Crawl-delay extension.
Because crawler policies evolve, the safest implementation is to link the site’s governance documentation to the current vendor documentation and review it rather than hard-coding assumptions into a permanent editorial article.
Perplexity Crawler Governance
Perplexity documents PerplexityBot as a search crawler designed to surface and link websites in Perplexity search. The documentation explicitly distinguishes it from foundation-model training.
Perplexity also documents Perplexity-User for user-triggered access. Its current documentation states that because the fetch is user-requested, this fetcher generally ignores robots.txt.
Why This Matters
This is exactly why a generic statement such as “robots.txt controls all AI access” is too broad. The effective behavior depends on the operator, the agent and the purpose.
Perplexity also publishes IP information for its documented agents. When a site uses a WAF, Perplexity recommends combining user-agent and IP information when configuring allow rules.
Standards, Proposals and What to Trust
robots.txt and RFC 9309
RFC 9309 is the standards-track Robots Exclusion Protocol specification. It defines the syntax, user-agent groups, Allow/Disallow matching, redirects, parsing and security considerations.
For consequential crawler-access decisions, RFC 9309 should be treated as the protocol foundation, while operator documentation determines vendor-specific behavior.
llms.txt
llms.txt is frequently presented as an AI-specific discovery or content-management mechanism. It should not, however, be treated as a universal replacement for robots.txt. Adoption and interpretation are not equivalent to the standardized robots exclusion protocol.
A sensible editorial policy is to monitor llms.txt developments while keeping robots.txt and normal technical SEO as the primary operational controls.
Emerging AI Preference Proposals
The broader ecosystem includes proposals and experiments around machine-readable AI preferences, content signals and agent behavior. These should be labeled clearly as proposals unless a specific operator has documented production support.
Evidence rule: Separate three categories in every crawler policy: standardized behavior, vendor-documented behavior, and community proposals. Never present a proposal as an enforced web standard.
How to Verify a Crawler Is Genuine
A user-agent string is not cryptographic proof of identity. A request can claim to be one crawler while originating from an unrelated system. Therefore, serious crawler governance should combine multiple signals.
Verification Layer 1: User-Agent
Start with the user-agent string. It provides a useful first classification and can help identify which vendor’s policy should be consulted.
Verification Layer 2: Published IP Ranges
Where an operator publishes IP ranges, compare the request origin against those published ranges. Perplexity, for example, publishes IP endpoints for its documented agents and recommends using current IP information for WAF configuration.
Verification Layer 3: Reverse DNS Where Appropriate
Where the operator documents an appropriate reverse-DNS verification method, it can provide another signal. DNS should never be treated as the only authentication mechanism unless the vendor explicitly documents the procedure.
Verification Layer 4: Server Logs
Compare the claimed agent, source address, requested URL, status code, response size, frequency and timing. A crawler that repeatedly claims one identity while originating from unrelated infrastructure deserves investigation.
Verification Layer 5: Vendor Documentation
Finally, compare observed behavior with the vendor’s current crawler documentation. The documentation is the authoritative reference for the meaning of the agent and the controls it honors.
Server Logs and WAF Verification
robots.txt tells a compliant crawler what the publisher requests. Server logs tell the publisher what actually happened. That difference makes logs essential for technical SEO governance.
Minimum Log Fields
- Timestamp
- Requested URL
- HTTP method
- Status code
- User-agent
- Source IP
- Response size
- Response time
- Referrer where available
WAF Can Override Your Intended Outcome
A robots.txt rule can allow a crawler while a CDN or WAF blocks its actual request. The opposite can also occur: a request can reach the origin even though the publisher intended to disallow it if the requesting system does not honor robots.txt.
This creates a useful diagnostic model: policy layer → network layer → application layer → content layer. All four should be evaluated independently.
Do Not Whitelist User-Agent Alone
For security-sensitive WAF configuration, user-agent-only allow rules can be spoofed. Where the vendor publishes authoritative IP information, combine the signals according to the vendor’s own recommendations. Perplexity explicitly recommends combining user-agent and IP address conditions for WAF rules.
AI Crawler Governance Framework
A durable AI crawler policy should not be a random collection of robots.txt lines. It should be a documented governance system.
1. Maintain an Agent Inventory
Create a central record of every known AI/search agent relevant to the organization. For each agent record the operator, purpose, user-agent token, documentation URL, IP source if published, and last verification date.
2. Define Policy by Purpose
Separate training, search and user-triggered retrieval. This avoids broad policies that accidentally remove useful search visibility.
3. Define Content Classes
Not every page has the same business value. Consider categories such as public editorial content, premium content, user-generated content, commercial pages, administrative paths and private resources.
4. Keep Security Separate
Do not use robots.txt to protect private information. Use authentication, authorization, application controls and appropriate network security.
5. Review Changes
Crawler ecosystems change quickly enough that an annual review can be too slow for high-value sites. A quarterly governance review is a reasonable operational starting point, with immediate review when a major vendor announces a new crawler or policy.
6. Document Every Change
- What changed?
- Which agent changed?
- Why was it changed?
- Which pages are affected?
- What evidence supported the decision?
- Who approved it?
- When should it be reviewed again?
Governance Example
| Field | Example |
|---|---|
| Operator | Example AI provider |
| Agent | ExampleBot |
| Purpose | Search indexing |
| Policy | Allow |
| Affected paths | Public editorial content |
| Evidence | Current vendor documentation |
| Verified | Record actual verification date |
| Owner | Technical SEO / Web Operations |
| Next review | Set an actual review date |
Troubleshooting Common AI Crawler Problems
“I blocked the training crawler, but my site is still appearing in AI search.”
This can be expected. Training access and search access may be controlled by separate agents. Blocking one does not necessarily remove the other.
“My robots.txt allows the crawler, but the crawler receives 403.”
Check the CDN, WAF, origin firewall, rate limits, bot-management rules and application security controls. robots.txt is only one layer.
“The crawler is requesting pages I disallowed.”
First verify that the observed request is actually from the claimed operator. Then check the exact robots.txt version available to the crawler, the user-agent token, group matching, path specificity, caching and whether the operator documents exceptions.
“My rule works for one AI bot but not another.”
That is normal. Different operators can use different user-agent tokens and can implement different retrieval models. Check the operator-specific documentation.
“I changed robots.txt but behavior has not changed.”
Crawler systems may cache robots.txt or may take time to observe configuration changes. Vendor documentation should determine expected propagation behavior. Perplexity, for example, states that robots.txt changes can take up to 24 hours to be reflected in its systems.
“A crawler is claiming to be a major AI company.”
Do not trust the user-agent string alone. Cross-check the IP against published vendor ranges where available and inspect server logs.
Use the diagnostic sequence to separate policy problems from network, WAF, identity and application problems.
AI Crawler Access Checklist
Use this checklist before publishing a crawler policy or making a major robots.txt change.
- Identify the exact crawler or product token.
- Confirm its purpose from current first-party documentation.
- Separate training, search and live retrieval decisions.
- Confirm whether the operator honors robots.txt.
- Check whether the operator documents special behavior.
- Use the narrowest practical user-agent rule.
- Verify Allow/Disallow path specificity.
- Do not use robots.txt as a security mechanism.
- Check CDN and WAF behavior.
- Verify published IP ranges where available.
- Review server logs after implementation.
- Document the policy owner.
- Record the evidence source.
- Record the actual verification date.
- Set a future review date.
- Update the policy when a vendor changes its crawler ecosystem.
Technical SEO Publication Checklist
- One clear H1.
- Logical H2/H3 hierarchy.
- Descriptive title and meta description.
- Canonical URL configured in the CMS.
- Breadcrumbs use live internal links.
- Related articles use live internal links.
- Tables have captions and scoped headers.
- SVGs have accessible titles and descriptions.
- Images, if added, have meaningful alternative text.
- Page is mobile responsive.
- CSS is scoped to the article component.
- JSON-LD contains real production values.
- No placeholder dates, URLs or organization data.
- All time-sensitive claims have a source and review process.
