Both control how search engines treat your pages, and both use similar-looking words like “disallow” and “noindex.” That surface resemblance is exactly why they get confused, and the confusion produces the most common robot-control mistake on the web: a page that was supposed to disappear from search stays visible, or a page that should rank quietly drops out. The two tools live on different layers of the search pipeline. Once that split is clear, choosing between them stops being guesswork.
robots.txt controls crawling, not indexing #
robots.txt is a plain text file that lives at the root of a domain, for example at example.com/robots.txt. It follows the Robots Exclusion Protocol, one of the oldest conventions on the web, dating to 1994. Cooperative crawlers read it before fetching anything else on the site. Non-cooperative bots and scrapers ignore it, because the protocol is advisory rather than enforced.
A minimal file looks like this:
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /api/public/
Sitemap: https://example.com/sitemap.xmlUser-agent names which crawler the rules apply to, with * meaning all crawlers. Disallow lists paths the crawler should skip. Allow carves an exception out of a broader Disallow. Sitemap points to the XML sitemap. Some crawlers honor extensions like wildcards (* for any sequence, $ for end-of-URL) or Crawl-delay, but support varies and those four cover the core.
The point most people miss is what robots.txt actually governs. It governs crawling, meaning whether a crawler is allowed to fetch a URL at all. It does not govern indexing. Google’s own documentation is explicit that robots.txt “is not a mechanism for keeping a web page out of Google.” A URL disallowed in robots.txt can still appear in search results, without a description, if Google discovers the URL from links on other sites. The block prevents the fetch, not the listing.
Meta robots controls what happens after the page is fetched #
The meta robots tag is an HTML element placed in the <head> of an individual page. It tells crawlers how to treat that specific page once they have fetched it.
<head>
<meta name="robots" content="noindex, nofollow">
</head>The name="robots" attribute targets all crawlers, or a single bot if named, such as name="googlebot". The content attribute lists the directives, comma-separated. The four most common are straightforward. noindex keeps the page out of the search index. nofollow tells the crawler not to follow links on the page. noarchive prevents a cached copy from showing. nosnippet removes the description text from search results. Others exist, including noimageindex, unavailable_after, and max-snippet, but those first four cover most real use cases.
The consequential detail is timing. A meta robots tag is read only when a crawler actually fetches the page. If robots.txt has already blocked the page, the crawler never arrives, and the meta tag is never seen. The directive simply never applies.
Crawl controls discovery, index controls visibility #
The cleanest way to keep the two straight is to separate the two jobs a search engine does. Crawling is fetching a page to read its content. Indexing is deciding to store the page and consider it for search results.
robots.txt sits at the crawl layer. It grants or denies permission to fetch a URL. Meta robots and the X-Robots-Tag header sit at the index layer. They grant or deny permission to include a page in search results after it has been fetched.
This layer split explains why noindex does not work inside robots.txt. Google officially dropped support for a noindex directive in robots.txt on September 1, 2019, after retiring the code that handled unsupported, unpublished rules. The reasoning follows directly from the layers: noindex is an indexing instruction, and robots.txt operates before indexing happens. Putting it there was always semantically wrong, even though some crawlers honored it informally for years. Bing never supported it either. If you still have Noindex: lines in a robots.txt file, they are now silently ignored.
The practical consequence is sharp. If a page needs to stay out of search, robots.txt alone will not do it, and using robots.txt to block the page actively prevents Google from seeing the meta noindex that would. The page gets crawled less, the noindex never gets read, and the URL still surfaces because an external link pointed to it. The crawl block produces the opposite of the intended result.
Where they live and what they affect #
| Aspect | robots.txt | Meta robots tag |
|---|---|---|
| Location | Root directory (<!–INLINECODE27–>) | <!–INLINECODE28–> of each HTML page |
| Scope | Whole site or path patterns | One page at a time |
| Format | Plain text file | HTML <!–INLINECODE29–> element |
| Layer | Crawl control | Index control |
| File types | Any URL pattern | HTML pages only (non-HTML uses X-Robots-Tag) |
| Visibility | Public, anyone can read it | Visible in page source |
| Granularity | Path patterns, wildcards, user-agent targeting | Per-page directives |
| Discovery | First file every crawler reads | Read when the page is crawled |
A site uses robots.txt for broad rules: block an entire admin section, disallow query-parameter combinations that generate duplicate views, point crawlers to the sitemap. Its strength is reach, since one entry controls thousands of URLs. A site uses meta robots for individual pages: a post-submission thank-you page with no search value, an unfinished test page, a staging environment where every page needs noindex. Meta robots covers the granularity robots.txt cannot reach without becoming unmanageable.
X-Robots-Tag: the header most articles skip #
A full picture needs a third element. The X-Robots-Tag is an HTTP response header that does the same job as the meta robots tag, but travels in the response rather than inside the HTML.
HTTP/1.1 200 OK
X-Robots-Tag: noindex, nofollowAny directive supported by meta robots works here, the whole vocabulary. The difference is what it can control. Meta robots requires an HTML <head>, so it works only on HTML pages. PDFs, images, videos, JSON files, and plain text files have no head, so the meta tag cannot apply to them. X-Robots-Tag has no such limit, because an HTTP response is sent for every resource type the server delivers. A common implementation is an Apache .htaccess directive or an Nginx nginx.conf block that adds X-Robots-Tag: noindex to every PDF on the site, keeping them out of search without editing each file or listing each one in robots.txt.
Conflicts are usually contradictions in disguise #
When the two tools are used together, a handful of conflict patterns recur.
The most common is robots.txt blocking a page that carries a noindex meta tag. The intent is “hide this from search.” The result is the reverse: Google cannot crawl the page, so it never reads the noindex, and the URL still appears from external links, without a description. The fix is to remove the robots.txt block, let Google crawl the page, and let the meta noindex do its work. In Google Search Console this shows up as the “Indexed, though blocked by robots.txt” status, which is a direct signal of this exact mistake.
A second pattern is combining noindex with a canonical tag pointing elsewhere. The signals contradict: noindex says drop this page, while canonical says treat another page as the authoritative duplicate. Google’s handling here is not fully predictable, so pick one mechanism based on the actual goal. For a genuine duplicate, use canonical alone. For a page that should not be in search at all, use noindex alone.
A third pattern is contradictory directives in a single tag, such as index, noindex or follow, nofollow. With X-Robots-Tag the more restrictive rule wins; with meta robots the outcome depends on the crawler. The fix is simply not to write conflicting directives.
How to check what crawlers actually see #
Diagnosis starts with what crawlers see, not what the config intends. Three tools cover most cases. Google Search Console’s URL Inspection reports whether a URL is indexed, which robots directives Google detected, and whether the page is blocked by robots.txt. Second, browser DevTools or curl -I reveals the full HTTP response headers, including any X-Robots-Tag, which is the only way to verify non-HTML resources that have no viewable meta tag. Third, a site crawler like Screaming Frog or Sitebulb inventories every page’s indexability across the whole site, which is the only practical way to catch patterns rather than single instances on a large site.
Frequently asked questions #
Does disallowing a URL in robots.txt remove it from Google?
No. This is the single most persistent myth about the file. Disallow blocks crawling, not indexing. A blocked URL can still be indexed and shown, without a description, when other sites link to it. To keep a page out of search, use noindex (with crawling allowed) or password protection.
Can I put noindex in robots.txt?
Not effectively. Google dropped support for that directive on September 1, 2019, and Bing never supported it. Place noindex in a meta robots tag or an X-Robots-Tag header instead.
Why do I need to allow crawling to use noindex?
Because a crawler has to fetch the page to read the noindex directive. Blocking the page in robots.txt keeps the crawler away, so the noindex is never seen and the page can remain indexed. Allow the crawl, and let the noindex remove the page.
How do I keep PDFs and images out of search?
Meta robots needs an HTML head, which non-HTML files lack. Use X-Robots-Tag: noindex in the HTTP response, commonly set once in the server config for all files of a type.
Is robots.txt or noindex enough to protect private content?
No. Both are advisory and honored only by cooperative crawlers. For genuinely private content, use authentication. Use noindex for content that is fine to access but should not surface in search.
The short version: crawl problems get crawl tools, and index problems get index tools. Ask whether the URL should be reachable by crawlers at all. If not, disallow it in robots.txt. If it should be reachable but not appear in search, use noindex (and make sure nothing in robots.txt blocks the crawl). If it is a non-HTML file that should not appear in search, use X-Robots-Tag. Matching the tool to the layer the problem lives on is what keeps pages from doing the opposite of what you intended.