Search bots
Index pages so AI assistants can find and cite them in answers. Block these and you disappear from AI search.
- OAI-SearchBot (OpenAI)
- Claude-SearchBot (Anthropic)
- PerplexityBot (Perplexity)
- Amazonbot (Amazon)
Free AI crawler checker · no signup
Can ChatGPT read your site?
Check whether the crawlers behind ChatGPT, Claude, Perplexity and Gemini can get in, from your robots.txt to your firewall.
Checks 14 AI crawlers, your firewall, robots directives and whether your content is in the HTML.
Checking AI crawler access… Loading your page Most sites take a few seconds.
THREE KINDS OF AI CRAWLER
Not every AI bot is the same. Block the right ones.
User bots
Fetch a page the moment someone asks an assistant about it, like a browser would. Block these and assistants can’t read the link they were given.
- ChatGPT-User (OpenAI)
- Claude-User (Anthropic)
- Perplexity-User (Perplexity)
Training bots
Gather pages to train future models. Blocking them is a reasonable choice, and doesn’t keep you out of AI answers.
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- CCBot (Common Crawl)
- Bytespider (ByteDance)
THE CRAWLERS WE CHECK
14 AI user agents, and what each one does.
GPTBot· OpenAI- Collects pages to train future GPT models.
OAI-SearchBot· OpenAI- Indexes pages for ChatGPT search results.
ChatGPT-User· OpenAI- Fetches a page when someone asks ChatGPT about it.
ClaudeBot· Anthropic- Collects pages to train Claude models.
Claude-SearchBot· Anthropic- Indexes pages for Claude’s web search answers.
Claude-User· Anthropic- Fetches a page when someone asks Claude about it.
PerplexityBot· Perplexity- Indexes pages for Perplexity answers.
Perplexity-User· Perplexity- Fetches a page when someone asks Perplexity about it.
Google-Extended· Google- Controls whether Gemini models may train on your pages. Google Search and AI Overviews use Googlebot instead.
Applebot-Extended· Apple- Controls whether Apple Intelligence may train on your pages.
Meta-ExternalAgent· Meta- Collects pages to train Meta AI models.
Amazonbot· Amazon- Crawls pages for Alexa and Amazon’s AI answers.
CCBot· Common Crawl- Builds the open Common Crawl dataset that many AI models train on.
Bytespider· ByteDance- Collects pages for ByteDance’s AI models.
ONE PAGE IS A START
SEOgent checks every page, every week.
It connects to Google Search Console, audits your whole site, ranks the fixes worth doing, and hands them to Claude Code, Codex or Cursor over MCP.
Request an invite- ✓ Every page, re-checked daily
- ✓ Clicks, rankings and index status from Search Console
- ✓ A ranked fix list with step-by-step plans
- ✓ Core Web Vitals for your most visited pages
- ✓ Keyword gaps against your competitors
- ✓ An MCP server your coding agent works through
FAQ
About AI crawlers.
Is the AI crawler checker free?
Yes. No signup, no email. Enter a URL and the result appears in a few seconds. Each visitor can run a few checks an hour.
Should I block AI crawlers?
It depends on what you want. Blocking training bots like GPTBot or CCBot keeps your pages out of future models but doesn’t stop ChatGPT or Perplexity citing you. Blocking search and user bots like OAI-SearchBot, Claude-SearchBot or PerplexityBot keeps you out of AI answers altogether. Most businesses that want to be found should allow the search and user bots.
How do I let AI crawlers into my site?
Remove any “Disallow: /” lines under their user agents in robots.txt, or add a group like “User-agent: OAI-SearchBot” with “Allow: /”. If you use Cloudflare, check Security → Bots: its AI bot blocking turns AI crawlers away at the firewall, whatever robots.txt says.
What does the firewall check do?
It asks for your page a second time as GPTBot. If a browser gets the page but GPTBot gets an error, a firewall or CDN is probably blocking AI crawlers. It’s a hint rather than proof, because some firewalls only let the real crawler through from its own addresses.
Do I need an llms.txt file?
No. llms.txt is a proposed file that summarises a site for AI tools. None of the major AI crawlers require it, so treat it as optional. Getting robots.txt right and putting your content in the HTML matter far more.
Why does “content in the HTML” matter?
AI crawlers like GPTBot and ClaudeBot don’t run JavaScript. If your page draws its content with JavaScript, as many AI-built and single-page apps do, they see an almost empty page, even when robots.txt lets them in.