Robots.txt Blocking Pages
A misconfigured robots.txt file can prevent search engines from crawling and indexing important pages, effectively making your content invisible in search results. Common mistakes include using overly broad Disallow rules, blocking CSS/JS files needed for rendering, or accidentally blocking the entire site with "Disallow: /".
Síntomas
- Pages not appearing in Google search results despite being published
- Google Search Console showing "Blocked by robots.txt" errors
- Googlebot unable to render pages due to blocked CSS/JavaScript
- Sitemap URLs conflicting with robots.txt disallow rules
Causas comunes
- Overly broad Disallow rules (e.g., Disallow: /)
- Staging robots.txt deployed to production
- Blocking CSS and JavaScript files needed for page rendering
- Wildcard patterns matching unintended URLs
- Rules targeting wrong user-agents
Solución paso a paso
- 1
Test robots.txt rules
Analyze your robots.txt file and test specific URLs against the rules.
- 2
Check HTTP headers
Verify robots.txt is served with correct Content-Type and status code.
Usar HTTP Headers → - 3
Check site accessibility
Verify the pages themselves are accessible when not blocked.
- 4
Fix robots.txt rules
Remove or modify overly restrictive Disallow directives. Allow Googlebot access to CSS and JavaScript files.
- 5
Request recrawl
After fixing, request recrawl through Google Search Console or wait for the next scheduled crawl.
Herramientas de diagnóstico recomendadas
Monitor Robots.txt
Get alerts if robots.txt changes unexpectedly.