AI Crawler Accessibility Checker
Enter any URL to check whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and Bytespider can access your page. This tool checks your robots.txt rules, X-Robots-Tag response headers, and noindex meta tag in one pass.
Why AI Crawler Access Matters for Your Site
AI-generated answers on Perplexity, ChatGPT with web search, and Google AI Overviews are increasingly where users get answers to questions — and where they discover products. When a user asks "what is a good indie product directory" or "what tools should I use to check my site's SEO", the AI systems that answer those questions have retrieved and indexed pages from across the web. If your pages are blocked to those crawlers, your content cannot be cited.
Many site owners accidentally block AI crawlers through a misconfigured robots.txt, an overly broad X-Robots-Tag header, or a noindex meta tag applied to pages they intend to be public. This tool checks all three blocking mechanisms in one request so you can identify which, if any, is preventing AI visibility for your important pages.
AI Crawlers Checked
Three Ways AI Crawlers Get Blocked
robots.txt
The most common source of accidental blocks. A Disallow: / rule under User-agent: * or under a specific bot's user-agent blocks all access. A common mistake is adding a wildcard block for privacy without exempting Googlebot and AI retrieval crawlers.
X-Robots-Tag response header
A noindex or none value in the X-Robots-Tag HTTP header instructs crawlers not to index the page. If your server or CDN (e.g. Vercel edge config, Cloudflare rules) returns this header for pages you want indexed, crawlers will skip them.
Meta robots noindex tag
A <meta name='robots' content='noindex'> tag in the HTML head tells crawlers not to index the page. Common causes: a staging environment setting left on production, a CMS default, or a plugin that adds noindex to non-canonical pages.
How to Allow AI Crawlers in Next.js robots.ts
// app/robots.ts
import { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
// Allow all crawlers by default
{ userAgent: '*', allow: '/' },
// Block AI training while allowing AI retrieval
{ userAgent: 'Google-Extended', disallow: ['/'] },
// Or allow everything including AI training:
// { userAgent: 'Google-Extended', allow: '/' },
],
sitemap: 'https://yoursite.com/sitemap.xml',
}
}Frequently Asked Questions
What is the difference between blocking AI training crawlers and AI retrieval crawlers?
AI training crawlers (like CCBot, Google-Extended, and GPTBot in training mode) collect your content to include in datasets used to train large language models. Blocking them prevents your content from being used in future model training but has no effect on current AI-generated answers. AI retrieval crawlers (like PerplexityBot and the retrieval component of GPTBot) index your content for real-time web search in tools like Perplexity and ChatGPT with web search. Blocking retrieval crawlers prevents your content from being cited in AI-generated answers today. These are distinct use cases, and you can block one while allowing the other using specific user-agent rules in robots.txt.
If I block GPTBot in robots.txt, will my site still appear in ChatGPT answers?
It depends on which component is blocked. GPTBot is the user-agent for OpenAI's training and real-time retrieval crawler. If you block GPTBot entirely in robots.txt, your content will not be crawled for ChatGPT's web search feature, which means pages blocked from GPTBot will not appear as sources in ChatGPT's web-search-enabled responses. However, ChatGPT's base knowledge (the model weights) was trained on data collected before you added the block, so older content from your site may still inform model responses indirectly without citation.
What happens if my page has a noindex tag but no robots.txt block for AI crawlers?
A noindex meta tag tells compliant crawlers not to include the page in their search index, but the page can still be visited and its content can still be read. AI training crawlers may still fetch the page and use its content for training datasets even if it is noindexed, because they are not building a search index in the traditional sense. To prevent AI crawlers from reading the content of a noindexed page entirely, add an explicit X-Robots-Tag header or robots.txt Disallow rule for the specific AI crawler user-agents.
How does Google-Extended differ from Googlebot?
Googlebot is Google's primary crawler for Search indexing. Google-Extended is a separate crawler Google introduced in 2023 specifically for collecting training data for Gemini and other Google AI products. You can block Google-Extended to prevent your content from being used in Google AI training while keeping Googlebot access and maintaining your search rankings. These are completely independent — blocking one does not affect the other. To block only AI training: add 'User-agent: Google-Extended' with 'Disallow: /' to your robots.txt.
Does blocking AI crawlers hurt SEO?
Blocking AI training crawlers (Google-Extended, CCBot, GPTBot for training) has no impact on your search rankings. Googlebot is a separate user-agent and is not affected by rules targeting those crawlers. However, blocking AI retrieval crawlers (PerplexityBot, the retrieval component of GPTBot) does reduce your chances of appearing as a cited source in AI-generated answers on those platforms. This is a tradeoff between controlling how your content is used versus visibility in AI search surfaces. Many publishers block training use while allowing retrieval.