A developer ships a cleanup on a Friday. The old staging directory is finally gone, so they add one line to the site’s robots.txt: Disallow: /. It was written for the staging subdomain. It landed on production. Nothing 404s, nothing looks broken to a visitor, and the deploy passes every check the team has. Over the next few crawl cycles Googlebot reads that single line, obeys it, and stops fetching the whole site. The traffic drains away over the following weeks, and the cause is four characters in a plain text file most teams open once and never read again.
robots.txt is not a suggestion box. It is a small set of instructions that Google’s crawler runs against every URL before it decides whether to fetch. Since September 2022 the format has been a published internet standard (RFC 9309, the Robots Exclusion Protocol), and Google documents exactly how its crawlers read the file. The rules are precise. Most of the expensive mistakes come from guessing at them instead of knowing them.
What the file actually decides, and what it does not #
robots.txt controls crawling, which is whether Googlebot fetches a URL. It does not control indexing, which is whether a URL can appear in results. Those two live on different layers, and the gap between them is where sites get hurt. Block a URL in robots.txt and Google will not fetch it, but if another page links to that URL, Google can still list it in search with no description and the label “Indexed, though blocked by robots.txt.” The block stopped the crawl, not the listing.
This has a sharp practical consequence. If the goal is to keep a page out of results, robots.txt is the wrong instrument, because blocking the crawl also blocks Google from ever reading the noindex rule that would have removed the page. Crawl control and index control answer different questions, and reaching for the first when you need the second is one of the most common self-inflicted indexing problems.
How Googlebot finds the group of rules that applies to it #
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines, then the Allow and Disallow rules under it. When Googlebot reads the file, it does not obey every group. It finds the group with the most specific user-agent line that matches its own name, and follows only that one.
So a file with a User-agent: * group and a separate User-agent: Googlebot group means Googlebot ignores the wildcard group entirely and uses the Googlebot group. The catch is that the specific group must then be complete: rules do not fall through from * to fill gaps. If several groups name the same crawler, Google merges their rules into one group internally. The lesson that trips people up is the first half: a targeted group silently switches off the general one, so any rule you assumed was inherited from * is simply gone for that crawler.
The rule that decides which line wins #
Inside the group, order does not matter. Google does not read the rules top to bottom and stop at the first match. Instead it evaluates every matching rule and applies the most specific one, measured by the length of the rule path. The longer, more specific path wins. That is the opposite of what most people assume, and assuming the file runs top to bottom is where a lot of confident edits go wrong. When two conflicting rules match with the same path length, Google uses the least restrictive one, which means Allow beats Disallow on a tie.
Two worked cases make the mechanic concrete:
| Requested URL | Matching rules | Verdict | Why |
|---|---|---|---|
/page |
Allow: /p and Disallow: / |
Crawled | /p is the longer, more specific path, so it wins |
/folder/page |
Allow: /folder and Disallow: /folder |
Crawled | Equal path length, so the least restrictive rule (Allow) wins |
Because precedence is length-based rather than order-based, adding a broad Disallow: / at the bottom of a file does not override the specific Allow rules above it. The file is not a script that runs in sequence. It is a set of rules that Google resolves by specificity every time.
What the wildcards actually mean #
Two special characters do most of the pattern work, and both are Google extensions to the base standard. The asterisk * matches zero or more of any character. The dollar sign $ anchors a rule to the end of the URL. A trailing * is redundant and Google ignores it, so /fish* and /fish behave identically.
Disallow: /*.pdf$blocks every URL that ends in.pdf, and only those, because$pins the match to the end.Disallow: /*?blocks any URL that contains a question mark, a blunt way to keep parameter and search URLs out of the crawl.Disallow: /private*/blocks/private/,/private-area/, and anything else that starts with/privateand contains a later slash.Allow: /*.css$andAllow: /*.js$are worth keeping explicit, so a broad directory block never accidentally starves Google of the files it needs to render the page.
Paths are case-sensitive. Disallow: /Admin does nothing to a URL served at /admin. The directive names themselves (user-agent, allow, disallow) are case-insensitive, but the paths after them are matched exactly as written.
Where the file lives, and what it covers #
A robots.txt file only governs the host, protocol, and port it sits on. It must live at the root of that host, at /robots.txt, and a file in a subfolder is not read. This scope rule breaks in three predictable ways. The file on https://example.com/robots.txt does not cover http://example.com, because the protocol differs. It does not cover https://shop.example.com, because that subdomain needs its own file. And www and non-www count as separate hosts, each requiring its own copy.
A team that migrates from HTTP to HTTPS, or spins up a subdomain, and forgets that the new host has no robots.txt of its own gets the default behavior for a missing file, which is the subject of the next section and is not always the safe one.
What Google does when the file is missing, broken, or oversized #
The response code Google gets when it requests robots.txt changes the outcome more than most teams expect, and one case is genuinely counterintuitive.
| robots.txt response | What Googlebot does |
|---|---|
| 2xx success | Reads and applies the file as written |
| 3xx redirect | Follows up to five redirects, then treats the file as not found |
| 4xx (404, 403, and similar), except 429 | Treats it as if no file exists, which means no crawl restrictions: Google may crawl everything |
| 429 or 5xx server error | Treats the whole site as temporarily disallowed; uses the last cached copy for up to 30 days if one exists, then falls back to no restrictions |
Read that last row twice. A 404 on your robots.txt tells Google to crawl the entire site, while a 503 on the same file tells Google to stop crawling the entire site. A server that returns a 5xx on robots.txt during an outage can pause crawling site-wide, which is fine for a short blip and a problem if it persists. Two more limits round this out: Google reads up to 500 kibibytes of the file and ignores anything past that, and it generally caches the file for up to 24 hours, so a fix does not take effect the instant you upload it.
The lines that quietly block the wrong thing #
A handful of patterns cause most of the real damage, and they share a trait: each looks reasonable until you know what it does.
- Blocking render resources. A rule like
Disallow: /wp-includes/or a broad block on a scripts folder can stop Google fetching the CSS and JavaScript a page needs. Under mobile-first indexing Google renders the page it crawls, so a page stripped of its layout files can be evaluated as a broken page. - The empty-value flip.
Disallow:with nothing after it allows everything.Disallow: /blocks everything. One character separates “crawl freely” from “crawl nothing,” which is exactly the mistake in the opening scene. - noindex in robots.txt. Some old guides put
Noindex:lines in the file. Google stopped honoring that on September 1, 2019, and it was never part of the standard. ANoindexline in robots.txt today does nothing. - crawl-delay. Google does not support
crawl-delay, and RFC 9309 left it out because real-world behavior was never consistent. A crawl-delay line has no effect on Googlebot. Google also removed its manual crawl-rate limiter tool in January 2024, so there is no dial to turn: Googlebot now adjusts its own rate based on how the server answers, backing off on its own when a site returns persistent server errors or gets slow to respond.
How much any of this matters for your site #
For a small brochure or local business site, robots.txt is often three or four lines that point to the sitemap and never need to change. The risk there is low and the temptation to over-manage the file is the bigger danger, because every edit is a chance to block the wrong path. A large site with faceted navigation, internal search, and millions of parameter URLs is the opposite case: there, disallowing the URL patterns that generate infinite low-value pages is one of the few levers that meaningfully shapes what Googlebot spends its time fetching.
The dividing line is not the file, it is the consequence of an error. On a ten-page site a bad rule costs a few pages. On a template-driven site of fifty thousand URLs, a single mispatterned rule can wall off an entire section, and the symptom shows up weeks later as a slow decline rather than an obvious break.
Treating the file like the code it is #
The safest habit is to treat robots.txt as production code rather than a config afterthought. Change one rule at a time. Before and after each change, paste the exact URL you care about into the robots.txt report or the URL Inspection tool in Google Search Console and confirm the verdict Google actually reaches, because reading the file with your own eyes is where the length-based precedence and wildcard behavior fool people. Keep the file boring, keep it small, and remember that the most damaging lines are rarely malicious ones. They are the well-meant rules written for one environment that end up running against the whole index.