Robots.txt SEO Guide: Rules That Actually Help
Your robots.txt file is a small text file with an outsized job: it tells crawlers where they may and may not go. Get it wrong and you can accidentally hide your whole site. Get it right and you steer crawl budget toward the pages that matter.
What robots.txt does (and doesn't)
It requests that compliant crawlers avoid certain paths. It does not guarantee secrecy, and it is not a security tool. A disallowed page can still be indexed if it is linked elsewhere.
Rules that actually help
- Allow your important content and your sitemap.
- Disallow low-value, infinite, or duplicate paths (faceted filters, internal search results).
- Reference your sitemap so crawlers find your URLs faster.
A common mistake
Blocking a page in robots.txt to keep it out of Google backfires. If you truly want a page out of the index, let crawlers reach it and use a noindex directive instead. Blocking it means Google never sees the noindex.
What is robots.txt?
Robots.txt is a plain-text file at your domain root that instructs search engine crawlers which URLs they are allowed to request.
FAQ
Where does robots.txt live? At the root of your domain, for example yoursite.com/robots.txt.
Can it hide pages from Google? No. To remove a page from search, use noindex, not a robots disallow.
