robots.txt Explained: A Practical Guide with Examples

Someone notices the staging site is showing up in Google. The obvious fix goes in within the hour: Disallow: / in staging’s robots.txt. Three weeks later the URLs are still in the results, now with a grey “No information is available for this page” underneath.

The block worked exactly as designed. It just did not do what everybody assumed it did — and that gap is the most expensive misunderstanding in SEO.

What robots.txt is and where it lives

robots.txt is a plain text file at the root of a host, served over HTTP:

https://example.com/robots.txt

Not /public/robots.txt, not /seo/robots.txt. Crawlers look in exactly one place and nowhere else.

It is scoped per host, protocol, and port. https://example.com, https://www.example.com, and https://shop.example.com each need their own file — which regularly surprises people whose staging environment sits on a subdomain. The format was eventually formalized as an internet standard (RFC 9309), so modern crawlers agree on the syntax far more than they once did.

It is a request, not a wall

Compliance is voluntary. Major search engines honour robots.txt. Scrapers, spam bots, and anything malicious simply ignore it. There is no enforcement mechanism.

The file is public. Anyone can read yoursite.com/robots.txt in a browser. So a line like:

Disallow: /admin-backup-2024/

is not hiding that directory — it is advertising it, in the first file any curious person opens.

Never use robots.txt as a security control. Private content needs authentication, IP restrictions, or simply not being on a public server. If it must not be seen, a text file politely asking robots not to look is not the answer.

The syntax

A robots.txt file is a series of groups. Each group starts with one or more User-agent lines and is followed by rules.

# Everything is welcome everywhere
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

• User-agent — which crawler the group applies to. * means “any crawler without a more specific group”. Crawlers obey the most specific matching group, and only that group.
• Disallow — a path prefix that should not be crawled. An empty value disallows nothing.
• Allow — carves an exception out of a broader Disallow. Where rules conflict, the more specific (longer) path generally wins.
• Sitemap — an absolute URL to your sitemap. Global, not tied to any user-agent group.
• # — a comment for the rest of the line.

Paths are matched as prefixes and are case-sensitive. Disallow: /Search will not block /search.

Wildcards

Two special characters are widely supported:

• * matches any sequence of characters.
• $ anchors the match to the end of the URL.

User-agent: *
Disallow: /*?sessionid=      # any URL containing that parameter
Disallow: /*.pdf$            # URLs ending in .pdf, but not /file.pdf?dl=1

One note on Crawl-delay: Google does not support it, though some other engines do. If crawl rate is a genuine server problem, handle it at the infrastructure level rather than hoping for cooperation.

Blocking crawling vs. preventing indexing

This is the part that costs people traffic, so read it twice.

robots.txt controls crawling. noindex controls indexing. They are different systems.

A URL blocked in robots.txt can still appear in search results. If other pages link to it, Google knows the URL exists — it just is not allowed to fetch it. So it may index the URL from those external signals alone, showing the bare link with no description. That is the result from the opening story.

Worse, the block is self-defeating if your goal was removal:

<!-- On the page you want out of the index -->
<meta name="robots" content="noindex">

Google can only see that tag by fetching the page. Block the URL in robots.txt and the crawler never fetches it, never sees the noindex, and the URL can linger indefinitely.

The correct pattern when you want a page out of search:

1. Allow crawling of the URL.
2. Serve noindex (meta tag or the X-Robots-Tag HTTP header, which also works for PDFs and images).
3. Wait for it to be recrawled and dropped.
4. Only then, if you also want to save crawl budget, add a robots.txt disallow.

Use robots.txt when you want to stop crawlers wasting time on URLs you do not care about — faceted filters, infinite calendars, internal search results. Use noindex when you want something gone from the results. Use real authentication when it must not be public at all.

Common recipes

Allow everything (the sane default):

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Block a staging or admin area:

User-agent: *
Disallow: /staging/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Block query-parameter noise but keep the clean pages:

User-agent: *
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?utm_

Block one specific bot while allowing the rest:

User-agent: BadBot
Disallow: /

User-agent: *
Disallow:

Remember that BadBot now follows only its own group — it inherits nothing from *.

The Robots.txt Generator assembles these rule sets and the sitemap line for you, which is a decent guard against typos in a file where one stray character has outsized consequences.

Dangerous mistakes

• Disallow: / shipped to production. The classic. A staging config gets promoted and the entire site drops out of search. If traffic falls off a cliff overnight, check this file first.
• Blocking CSS and JavaScript. Google renders pages to evaluate them. Disallowing /assets/ or /static/ means it sees an unstyled skeleton and may misjudge layout, mobile-friendliness, and anything that loads client-side.
• Assuming Disallow removes a page. It does not. See above.
• Forgetting subdomains. Blocking example.com does nothing for staging.example.com.
• Trailing-slash confusion. Disallow: /news blocks /news, /news/, and /newsletter. If you mean the directory, write /news/.
• Returning the wrong status. No robots.txt is fine — crawlers treat a 404 as “crawl everything”. A 500 is not, and can make crawlers back off entirely.

Test before you ship

• Fetch https://yourdomain.com/robots.txt in a browser and confirm plain text with a 200 status.
• Check every host you serve, including www and non-www.
• Use the robots.txt report in Google Search Console to see the version Google fetched and any parse errors.
• Test real URLs against your rules, especially ones near a wildcard.
• Verify the Sitemap line points at a file that exists and parses — the Sitemap Validator confirms that, and the Sitemap Generator will build one if you have none.

On size: Google documents a 500 KiB limit and ignores anything past it. Almost nobody hits this, but auto-generated files listing thousands of paths can.

Quick answers

Does robots.txt hide a page from Google? No. It stops crawling, not indexing. Use noindex to keep a page out of results.

Do I need a robots.txt file? Not required. A missing file means “crawl everything”, which is often what you want anyway. Add one when you need rules or want to declare a sitemap.

Can I block a page with both robots.txt and noindex? You can, but the block prevents the noindex from ever being seen. Allow crawling until the page is dropped.

Is robots.txt case-sensitive? The filename must be lowercase, and paths are case-sensitive. Directive names are not.

Will it stop scrapers? No. Only well-behaved crawlers honour it.

The takeaway

robots.txt is a direction sign for crawlers you already trust — useful for steering attention away from junk URLs, useless as a lock. Keep it small, keep CSS and JS crawlable, point it at your sitemap, and never confuse “do not crawl” with “do not index”. Then check it after every deploy, because the one time it goes wrong, it goes very wrong.