A misconfigured robots.txt file can wipe your site from Google overnight. This guide explains exactly how robots.txt works, what to block, what to allow, and the most common mistakes that silently kill SEO.
Check your site right now
Free SEO audit in 30 seconds — find all the issues covered in this guide.
Audit for free →https://yoursite.com/robots.txt — and tells search engine crawlers which pages they can and cannot access.\n\nIt sounds simple. It is deceptively dangerous.\n\nA single misplaced line in robots.txt can block Googlebot from crawling your entire site. Your pages vanish from search results within days. You won't get an error message. Google just quietly stops visiting.\n\nOn the flip side, a well-configured robots.txt improves your crawl efficiency, protects pages you don't want indexed, and keeps Google focused on your most important content.\n\nHere's everything you need to know.\n\n---\n\n## How robots.txt actually works\n\nWhen Googlebot (or any crawler) arrives at your site, it first fetches /robots.txt. The file contains a set of rules in this format:\n\n``\nUser-agent: *\nDisallow: /admin/\nAllow: /\n\nSitemap: https://yoursite.com/sitemap.xml\n`\n\nUser-agent specifies which crawler the rule applies to. * means all crawlers. You can target specific bots like Googlebot, Bingbot, or GPTBot.\n\nDisallow tells the crawler not to access that path. Disallow: /admin/ blocks everything under /admin/.\n\nAllow overrides a Disallow rule for a specific path. This is useful when you've blocked a directory but want to allow one file inside it.\n\nSitemap tells crawlers where to find your XML sitemap. You can list multiple sitemaps.\n\n### The matching rules you must understand\n\n- Rules are case-sensitive. /Admin/ and /admin/ are different paths.\n- A trailing slash matters. Disallow: /blog blocks /blog but not /blog/post-1. Disallow: /blog/ blocks everything under /blog/.\n- Disallow: with nothing after it means allow everything — it's not a typo, it's a valid way to reset rules.\n- The most specific rule wins when there's a conflict between Allow and Disallow.\n- robots.txt does not prevent a page from being indexed if another site links to it. It only blocks crawling.\n\n---\n\n## The 6 most damaging robots.txt mistakes\n\n### 1. Blocking your entire site\n\nThis is the most catastrophic error and it happens more often than you'd think — usually after a developer copies a staging robots.txt to production.\n\n`\n# DANGER — do not use on production\nUser-agent: *\nDisallow: /\n`\n\nThis blocks every crawler from every page. Your site disappears from Google within days. Check your robots.txt right now if you've recently migrated, launched a redesign, or moved from staging to live.\n\n### 2. Blocking CSS and JavaScript files\n\nOld SEO advice said to block CSS and JS to save crawl budget. That advice is wrong in 2026. Google renders JavaScript to understand your pages. If Googlebot can't fetch your CSS and JS, it sees a broken, unstyled page — and may rank it poorly or skip it entirely.\n\n`\n# Bad — don't do this\nDisallow: /wp-content/themes/\nDisallow: /wp-includes/\n`\n\nAllow Googlebot to access all your static assets unless you have a specific reason not to.\n\n### 3. Blocking pages that are in your sitemap\n\nIf a URL appears in your sitemap but is also blocked in robots.txt, you're sending Google a contradictory signal. Google will likely ignore the sitemap entry and not crawl the page.\n\nAlways cross-check: if it's in your sitemap, it shouldn't be in a Disallow rule.\n\n### 4. Confusing robots.txt with noindex\n\nrobots.txt and noindex meta tags do different things:\n\n- robots.txt Disallow prevents crawling. Google can't read the page.\n- noindex meta tag allows crawling but prevents indexing. Google reads the page but doesn't add it to search results.\n\nIf you want a page out of Google's index, use — not robots.txt. If you block with robots.txt, Google can't see your noindex tag, and the page may stay in the index via link discovery.\n\n### 5. Missing or wrong Sitemap directive\n\nMost sites miss the opportunity to declare their sitemap location in robots.txt. This is a quick win — it ensures every crawler that reads your robots.txt also finds your sitemap.\n\n`\nSitemap: https://yoursite.com/sitemap.xml\n`\n\nMake sure the URL is absolute (including https://), not relative.\n\n### 6. Blocking parameters that generate unique content\n\nSome sites block all URL parameters like ?page=2 or ?color=red to avoid duplicate content. But if those parameters generate distinct, useful content, blocking them removes that content from Google's index.\n\nUse Google Search Console's URL Parameters tool and canonical tags to handle duplicate parameter URLs — not a blanket robots.txt block.\n\n---\n\n## What you should actually block with robots.txt\n\nHere's what makes sense to block for most sites:\n\n`\nUser-agent: *\n\n# Admin and login pages — no SEO value, no need to crawl\nDisallow: /admin/\nDisallow: /wp-admin/\nDisallow: /login/\nDisallow: /account/\n\n# Search results pages — often duplicate content\nDisallow: /search/\nDisallow: /?s=\n\n# Checkout and cart — no SEO value\nDisallow: /cart/\nDisallow: /checkout/\n\n# Internal API endpoints\nDisallow: /api/\n\n# Staging or internal tools\nDisallow: /staging/\nDisallow: /internal/\n\nSitemap: https://yoursite.com/sitemap.xml\n`\n\nThis keeps Googlebot focused on pages that can actually rank, and avoids wasting crawl budget on admin interfaces and duplicate search results.\n\n---\n\n## Blocking AI crawlers with robots.txt\n\nrobots.txt has gained new relevance with AI training crawlers. If you don't want your content used to train AI models, you can block specific AI user-agents:\n\n`\n# Block OpenAI's training crawler\nUser-agent: GPTBot\nDisallow: /\n\n# Block Google's AI training crawler\nUser-agent: Google-Extended\nDisallow: /\n\n# Block Common Crawl\nUser-agent: CCBot\nDisallow: /\n`\n\nNote: these crawlers are supposed to respect robots.txt. Malicious scrapers won't. This is opt-out, not a guarantee.\n\n---\n\n## robots.txt SEO checklist\n\nRun through this before and after any site migration or launch:\n\n- [ ] Disallow: / is NOT present for User-agent: *` on production\n- [ ] CSS, JS, and image directories are not blocked\n- [ ] No URLs in your sitemap appear in Disallow rules\n- [ ] Admin, login, cart, and checkout paths are blocked\n- [ ] Sitemap URL is declared and uses an absolute https:// path\n- [ ] Search results and filtered parameter pages are blocked (if they're duplicates)\n- [ ] Pages you want removed from Google use noindex, not robots.txt\n- [ ] AI crawlers are blocked or allowed based on your preference\n\n---\n\n## How to check if robots.txt is hurting your rankings\n\nThe fastest check: open Google Search Console, go to Settings > robots.txt, and use the built-in tester. Paste any URL and see whether Googlebot is blocked or allowed.\n\nYou can also check your Coverage report for pages marked 'Crawl anomaly' or 'Blocked by robots.txt' — both indicate crawl issues you need to fix.\n\nFor a broader SEO audit that catches robots.txt issues alongside meta tags, Open Graph, schema markup, and more — run a free check at getmetafix.com. It takes 30 seconds, no account required, and it'll tell you exactly what's wrong.