7 Robots.txt Mistakes That Can Hurt Your Rankings

Robots.txt is a small file with outsized impact. It tells crawlers which paths they may request.

Mistake 1: Disallowing everything

Disallow: / blocks your entire site. This is often left over from a staging environment.

Mistake 2: Using robots.txt to hide pages from search

Blocked URLs can still be indexed if other sites link to them. Use a noindex meta tag instead, and leave the page crawlable.

Mistake 3: Blocking CSS and JavaScript

Google needs your assets to render pages properly.

Other mistakes

• Forgetting the Sitemap directive.
• Wrong file location — it must live at the root.
• Case-sensitive path typos.
• Conflicting Allow and Disallow rules.

Wrap up

Generate your file with a tool, then test it in Search Console's robots.txt report.