robots.txt Validator
Paste your robots.txt to validate syntax, check which crawlers are allowed or blocked, and catch mistakes that could accidentally de-index your entire site or leave AI training crawlers unrestricted.
What Is robots.txt?
robots.txt is a plain text file at the root of your domain (e.g. yoursite.com/robots.txt) that tells compliant web crawlers which pages they should and should not visit. It follows the Robots Exclusion Protocol, an informal standard that virtually all major search engines and many AI crawlers respect.
Every bot checks robots.txt before it crawls any page on your domain. A misconfigured file — even a single extra character — can accidentally block Googlebot from your entire site, preventing all your pages from appearing in search results. This is one of the most common and catastrophic technical SEO mistakes.
Most Dangerous robots.txt Mistakes
Disallow: /
Blocks all crawlers from your entire site. Googlebot will stop indexing everything. This is the single most common deployment accident — easy to commit, catastrophic to discover.
Missing User-agent: * rule
Without a wildcard rule, bots not listed explicitly may crawl with no restrictions at all. Always include a User-agent: * block even if it just says Allow: /.
Blocking /api/ routes needed for rendering
Next.js and other SSR frameworks sometimes request internal API routes during server-side rendering. Blocking these can break Google's ability to render and index your pages.
Using robots.txt for private content
robots.txt is publicly readable by anyone. Listing paths you want private tells attackers exactly where those paths are. Use authentication, not robots.txt, for genuinely private content.
No Sitemap directive
Without a Sitemap: directive, new crawlers and AI crawlers must guess where your sitemap is. Always declare the full sitemap URL in robots.txt.
AI Crawler User-Agents
Since 2023, AI companies deploy their own web crawlers to collect training data and power retrieval-augmented generation. These crawlers respect robots.txt. If you want to block AI training crawlers while keeping Google access, add explicit rules for each:
How to Generate robots.txt in Next.js
In Next.js App Router, create app/robots.ts to generate your robots.txt programmatically. Use an environment variable to serve different rules in production vs staging:
// app/robots.ts
import { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
const isProduction = process.env.VERCEL_ENV === 'production'
return {
rules: [
{
userAgent: '*',
allow: isProduction ? '/' : [],
disallow: isProduction ? ['/api/', '/admin/'] : ['/'],
},
// Block AI training crawlers but allow AI retrieval (optional)
{ userAgent: 'GPTBot', disallow: ['/'] },
{ userAgent: 'Google-Extended', disallow: ['/'] },
],
sitemap: 'https://yoursite.com/sitemap.xml',
}
}Frequently Asked Questions
Is robots.txt guaranteed to stop crawlers from accessing my pages?
No. robots.txt is a polite convention, not a technical barrier. Compliant crawlers like Googlebot, Bingbot, and GPTBot read and follow it. Malicious scrapers, data harvesters, and many other bots ignore it entirely. For pages you genuinely need to keep private — admin panels, user accounts, internal dashboards — use server-side authentication. robots.txt is only for managing which crawlers can access which public content.
What is the difference between robots.txt Disallow and a noindex meta tag?
They work at different stages. robots.txt Disallow prevents crawlers from visiting the page URL at all. A noindex meta tag (or X-Robots-Tag header) allows crawlers to visit the page but instructs them not to add it to their index. An important edge case: if you block a URL with robots.txt but another site links to it, Google may still show that URL in search results because it knows it exists from the backlink — it just cannot read the content. For reliable de-indexing, use noindex on a crawlable page, not robots.txt Disallow.
How do I block specific AI crawlers from training on my content?
Add specific User-agent rules for each AI crawler you want to block. The most common AI crawler user-agents are GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google AI training), CCBot (Common Crawl, used for many AI datasets), and PerplexityBot (Perplexity AI). Add a Disallow: / rule under each User-agent block to block that specific crawler from your entire site. You can also add a blanket block with User-agent: * and Disallow: / but this blocks all crawlers including Googlebot, which will destroy your search rankings.
Why should I add a Sitemap directive to robots.txt?
A Sitemap: directive in robots.txt tells every crawler — not just Googlebot — the exact URL of your sitemap. This is the fastest way for new crawlers and AI crawlers to discover all your content, since they check robots.txt on first visit. Without it, crawlers must guess your sitemap URL (usually /sitemap.xml) or find it through other signals. The directive accepts a full absolute URL: Sitemap: https://yoursite.com/sitemap.xml
Does Vercel automatically generate a robots.txt for Next.js apps?
No. Vercel does not generate a robots.txt automatically. You create one manually or generate it programmatically. In Next.js App Router, create app/robots.ts and export a default function that returns a MetadataRoute.Robots object. Next.js compiles this into a valid /robots.txt response at runtime. This approach lets you set different rules in development and production using environment variables, which prevents accidentally blocking Googlebot on a staging environment.