xseek Logo
xSeek/Docs
Back to documentation
Reference

The complete AI robots.txt guide.

Every AI crawler and user agent from the major companies, with copy-paste robots.txt configurations for each.

Updated Apr 202612 min readFor owners & developers

What is robots.txt?

A robots.txt file is a standard websites use to talk to crawlers and other automated clients. It states which parts of the site should not be processed or scanned.

With the rise of large language models, robots.txt gained a new job. It is now the main mechanism for controlling how AI systems interact with your content.

!
Important. Our tool always shows AI chatbots as allowed regardless of robots.txt directives, so you can see how AI systems might interact with your content.

AI crawlers by company

5 vendors

Each company runs its own agents, and they do not all do the same job. Training bots and search bots are worth treating differently.

OpenAI

OpenAI runs several crawlers for different jobs, from training their models to powering search inside ChatGPT.

GPTBot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot)

Trains GPT models on web content.

ChatGPT-User

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot)

Fetches pages when a user browses during a conversation.

OAI-SearchBot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)

Powers search results and browsing in ChatGPT.

Anthropic

Anthropic uses crawlers to support Claude, its conversational AI assistant.

Anthropic AI

Mozilla/5.0 (compatible; anthropic-ai/1.0; +http://www.anthropic.com/bot.html)

Training and improving Claude models.

ClaudeBot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Claude's web browsing capabilities.

Claude Web

Mozilla/5.0 (compatible; claude-web/1.0; +http://www.anthropic.com/bot.html)

Claude's web interface interactions.

Google

Google runs AI-specific crawlers alongside its traditional search crawlers.

Google-Extended

Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)

Trains Google's Gemini models.

Perplexity

Perplexity crawls the web to power its AI search product.

PerplexityBot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Perplexity's AI search.

xAI (Grok)

xAI uses crawlers to support its Grok assistant.

GrokBot

GrokBot/1.0 (+https://x.ai)

Training Grok.

Grok Search

xAI-Grok/1.0 (+https://grok.com)

Grok's search capabilities.

Grok DeepSearch

Grok-DeepSearch/1.0 (+https://x.ai)

Grok's advanced search.

Other major AI crawlers

Many other AI companies and search engines run their own agents.

Apple (Siri & Apple Intelligence)

Mozilla/5.0 (compatible; Applebot/1.0; +http://www.apple.com/bot.html)

Apple Extended

Mozilla/5.0 (compatible; Applebot-Extended/1.0; +http://www.apple.com/bot.html)

Meta (Facebook)

Mozilla/5.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Cohere

Mozilla/5.0 (compatible; cohere-ai/1.0; +http://www.cohere.ai/bot.html)

You.com

Mozilla/5.0 (compatible; YouBot (+http://www.you.com))

DuckDuckGo

Mozilla/5.0 (compatible; DuckAssistBot/1.0; +http://www.duckduckgo.com/bot.html)

Configuration examples

4 recipes

Your robots.txt belongs at the root of your site, for example https://example.com/robots.txt. Pick the recipe that matches your policy.

1. Allow all AI crawlers

robots.txt
User-agent: *
Allow: /

# Explicitly allowing each AI crawler is redundant with the wildcard above
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# ... and so on for other crawlers

2. Block all AI crawlers

robots.txt
# Allow regular crawlers
User-agent: *
Allow: /

# Block specific AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: GrokBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# ... and so on for other AI crawlers

3. Selective access per company

Allow the search bots that can cite you, block the ones that only train:

robots.txt
# Allow search bots but block training bots
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

4. Only specific directories

robots.txt
# Only allow AI crawlers into the public and blog sections
User-agent: GPTBot
Allow: /public/
Allow: /blog/
Disallow: /

User-agent: ClaudeBot
Allow: /public/
Allow: /blog/
Disallow: /

What allowing or blocking actually costs you

Benefits of allowing

  • Improved visibility in AI-powered search results
  • Your content can be used as a source in AI answers
  • Contributing to training may lead to better models
  • Stay relevant in an increasingly AI-driven web

Reasons to block

  • Protect proprietary or sensitive information
  • Prevent AI systems from reproducing your unique content
  • Reduce server load from crawler traffic
  • Keep control over how your content is used

Best practices

  • Review and update robots.txt as new AI crawlers appear
  • Be specific about which parts of the site each crawler can reach
  • Consider different policies for training bots versus search bots
  • Monitor server logs to see which AI crawlers actually visit
  • Verify your configuration with the robots checker
  • Stay informed about changes to AI crawler policies

Check your robots.txt

See how your file actually behaves against AI crawlers:

  • Analyze your robots.txt for AI crawler directives
  • See which AI crawlers are allowed or blocked
  • Get recommendations to improve the configuration
  • Track changes in AI crawler policies over time
Check your robots.txt →

FAQ

How do I block AI crawlers like GPTBot and Claude in robots.txt?

Add a user-agent directive for each bot. For example, "User-agent: GPTBot / Disallow: /" for OpenAI, and "User-agent: Claude-Web / Disallow: /" for Anthropic. This prevents those systems from crawling and training on your content.

Does robots.txt completely prevent AI models from accessing my content?

No. robots.txt is a voluntary protocol that ethical AI companies respect, but it cannot guarantee protection. Some systems may reach your content by other means. For sensitive content, add authentication and terms of service.

What are the main AI crawler user agents I should know about?

The major ones are GPTBot (OpenAI), Claude-Web (Anthropic), CCBot (Common Crawl) and GoogleBot (Google). Each represents a different company collecting web data for training and for serving products.

← Previous
ChatGPT ads & ranking playbook
Next →
OpenAI crawlers and user agents