A single misplaced line in one small text file has, more than once, made an entire website vanish from Google overnight. That file is robots.txt — and understanding it is one of those rare five-minute reads that can save you months of invisible SEO damage.
What Is Robots.txt?
Robots.txt is a plain text file placed at the root of your website (yoursite.com/robots.txt) that gives instructions to search engine crawlers about which parts of your site they are allowed, or not allowed, to crawl.
What Robots.txt Actually Does
It does not remove pages from Google’s index, and it does not guarantee privacy — it simply requests that well-behaved crawlers avoid certain paths. Its real job is managing crawl efficiency: keeping bots away from low-value areas (admin panels, internal search results, duplicate filter pages) so they spend more time crawling the pages that actually matter.
Robots.txt is a request, not a lock. Blocking a page from crawling does not reliably keep it out of Google’s index if other sites link to it — for true removal, use a “noindex” meta tag instead.
Basic Robots.txt Syntax
A simple robots.txt file might read: “User-agent: *” (applies to all crawlers), followed by “Disallow: /wp-admin/” (blocks that folder) and “Sitemap: https://yoursite.com/sitemap.xml” (points crawlers to your sitemap directly).
Common Robots.txt Mistakes
- Accidentally blocking the entire site with “Disallow: /” left over from a staging environment — one of the most common and costly SEO accidents.
- Blocking CSS or JavaScript files, which prevents Google from properly rendering and understanding the page.
- Using robots.txt to try to hide sensitive information — the file itself is publicly readable by anyone.
- Blocking a page in robots.txt while also including it in the sitemap — a direct contradiction.
- Forgetting to update robots.txt after a site migration or redesign.
How to Check Your Robots.txt File
Visit yoursite.com/robots.txt directly in a browser to see the live file, or use the robots.txt report inside Google Search Console to confirm Google is reading it correctly and identify any pages being unintentionally blocked.
Frequently Asked Questions
Can robots.txt hide a page from Google completely?
Not reliably. If other sites link to a blocked page, Google can still index the URL without crawling its content. Use “noindex” for guaranteed exclusion.
Does every website need a robots.txt file?
Not strictly, but having one gives you direct control over crawl behavior rather than leaving it entirely to default assumptions.
How do I know if my robots.txt is blocking something important?
Check the robots.txt report in Google Search Console, which flags exactly which URLs are being blocked and why.
Worried your robots.txt might be quietly blocking important pages? Request a free technical audit.