Robots.txt vs Meta Robots: Which Crawl Directive to Use

Robots.txt vs Meta Robots: the Short Answer

Robots.txt controls crawling. Meta robots controls indexing. They are not interchangeable, and the stronger-looking one is the weaker one for removing a page from Google.

Robots.txt is a single text file at the root of your site that tells crawlers which URLs they may fetch. A meta robots tag is an instruction on one page that tells search engines what to do with that page after it has been fetched. Because a crawler has to fetch a page to read its meta robots tag, blocking a URL in robots.txt makes any noindex on it unreadable — Google says so explicitly: if a page is disallowed from crawling, indexing rules on it "will not be found and will therefore be ignored".

So the decision is simple once you frame it as crawl vs index. To keep a page out of search results, use noindex and leave the URL crawlable. To stop crawlers spending requests on a section that has no business being fetched, disallow it in robots.txt. People search this as robots meta tag vs. robots.txt, as robots.txt vs meta tag, or as robots txt vs meta robots — same question, same answer, and the robots.txt vs noindex framing is the one that most often leads people to the wrong choice.

Last updated 11 August 2026. Rewritten and fact-checked against Google's current robots.txt specification and robots meta tag documentation; corrected eleven claims, including two references to a Search Console tool Google retired in 2023.

On this page

• The short answer
• Crawling and indexing are two different things
• How robots.txt actually works
• How meta robots tags actually work
• X-Robots-Tag: the third layer
• Robots.txt vs meta robots, side by side
• The de-index sequence most sites get backwards
• Mistakes that actually cost traffic
• Should I use robots.txt or noindex?
• Frequently asked questions

Crawling and Indexing Are Two Different Things

Nearly every robots.txt vs meta robots mistake traces back to treating one setting as two. The crawl vs index distinction is the whole game:

• Crawling is fetching. Googlebot requests a URL and downloads what comes back. Crawlability is about access.
• Indexing is filing. Google decides whether the fetched page is eligible to appear in results. Indexation is about eligibility.

Robots.txt operates on the first. Meta robots operates on the second. That is why a page can be blocked from crawling and still show up in search — Google's documentation confirms it "can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet." You have blocked the fetch, not the listing.

How Robots.txt Actually Works

Robots.txt is a UTF-8 text file at the root of a host. Crawlers request it before crawling, and Google supports exactly four fields: user-agent, allow, disallow and sitemap. Notably, crawl-delay is not supported by Google, whatever your plugin's UI suggests.

user-agent: *
disallow: /admin/
disallow: /*.pdf$
allow: /admin/public/

sitemap: https://example.com/sitemap.xml

Scope and matching rules

Three things about robots.txt are widely misunderstood:

• It is per host, protocol and port — not per domain. A file at https://example.com/robots.txt does not govern https://shop.example.com/, http://example.com/ or https://example.com:8181/. Every subdomain needs its own file. Crawlers never look in subdirectories for one.
• Path values must start with /. Only two wildcards exist: * matches zero or more characters, and $ anchors the end of the URL. So /*.pdf$ is valid and *.pdf$ is not. A trailing * is redundant — /fish* is simply /fish. Matching is case-sensitive.
• Precedence is by rule length, not by keyword. The most specific rule wins, measured by the length of the rule path — so disallow: /*.htm beats allow: /page for /page.htm. Only when two rules are genuinely equally specific does Google fall back to the least restrictive one. "Allow always beats Disallow" is a myth.

What robots.txt does not do

It does not remove pages from the index, and it is not security. The file is public at /robots.txt, so listing a private directory there advertises it. Use authentication for anything that actually matters.

Two operational limits are worth knowing. Google enforces a 500 KiB file size limit and ignores everything past it. And Google caches robots.txt for up to 24 hours — so a change is not instant. If you need it picked up sooner, use the Request a recrawl function in the Search Console robots.txt report.

Error handling matters too. A 4xx response (other than 429) is treated as though no robots.txt exists, meaning no crawl restrictions at all. A 5xx is the dangerous one: Google stops crawling the site for the first 12 hours, then falls back to the last good copy for 30 days. A flaky server returning 500s on /robots.txt can quietly suspend crawling of your whole site.

How Meta Robots Tags Actually Work

A meta robots tag is an HTML instruction giving page-level control over indexing and serving:

<meta name="robots" content="noindex">

Convention places it in the <head>, and that is still where it belongs — but Google explicitly states it "doesn't enforce placement of meta robots in the HTML head and will respect robots meta tags in the body section" too. Useful to know when auditing a template that injects tags late.

The name attribute targets a crawler. Google supports only two tokens here — googlebot and googlebot-news — and ignores anything else, so <meta name="bingbot" ...> does nothing at Google.

The rules Google supports — and three it ignores

The meta robots tags worth knowing: noindex (keep out of results), nofollow (don't follow this page's links), none (equivalent to noindex, nofollow), nosnippet, max-snippet, max-image-preview, noimageindex, notranslate, and unavailable_after: for content with a genuine expiry date. all is the default and has no effect when stated.

Three rules you will still see recommended are dead at Google: noarchive no longer does anything, because the cached-link feature it controlled no longer exists; nocache is unused; and nositelinkssearchbox is unused. If a checklist tells you to add them, that checklist has not been updated in years.

One subtlety worth internalising: nofollow in meta robots means "do not follow the links on this page." Without it, Google may use those links to discover the linked pages. It says nothing about whether this page gets indexed — a persistent confusion. Only noindex does that.

X-Robots-Tag: the Third Layer

Any rule you can put in a meta robots tag can instead be sent as an HTTP response header. That is the X-Robots-Tag, and it exists because you cannot put an HTML tag inside a PDF, an image, or a JSON endpoint.

X-Robots-Tag: noindex, nofollow

On Apache, deindexing every PDF on a site is a few lines in .htaccess:

<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>

The header can also target a user agent — X-Robots-Tag: googlebot: nofollow — which meta tags cannot do cleanly. Use it for non-HTML resources, or when you can reach server config but not page templates. For ordinary HTML pages prefer the meta robots tag: it is visible in page source and far easier to audit.

Note the precedence rule flips here. Among robots meta and X-Robots-Tag rules, the more restrictive rule wins — combine max-snippet:50 with nosnippet and you get nosnippet. That is the opposite of robots.txt path matching, where equally specific rules resolve to the least restrictive. Two layers, two opposite tie-breakers.

Robots.txt vs Meta Robots, Side by Side

The De-index Sequence Most Sites Get Backwards

This is the single most expensive error in the whole area. Blocking a URL in robots.txt first guarantees the noindex on it is never read, so the page can sit in the index as a bare URL listing indefinitely. The order that works:

1. Add noindex to the page, by meta robots or X-Robots-Tag, and leave the URL crawlable.
2. Wait for the recrawl and confirm the drop in the Search Console indexing statuses report.
3. Only then, if you also want to stop the crawl requests, add the robots.txt disallow.

Reverse those steps and you get the classic symptom: a page that will not index, or one that will not leave, where the cause is a directive conflict rather than anything to do with the content.

Mistakes That Actually Cost Traffic

Using robots.txt to remove a page from Google

The most common one, and now covered above. Disallow prevents fetching, not listing. Use noindex.

Leaving a staging noindex in production

A site launches with noindex inherited from staging and nothing indexes. Make removal an explicit release step, and check it on the live URL rather than trusting the CMS toggle.

Combining noindex with a canonical to another page

These are conflicting signals: one says drop this page, the other says consolidate it into a different one. Pick the behaviour you want — canonical tags for duplicates you want consolidated, noindex for pages that should not rank at all.

Blocking CSS and JavaScript

Google renders pages to index them. Disallowing the assets it needs to render means it judges a broken version of your page. If you block an /includes/ directory, add an allow for Googlebot.

Letting meta robots and X-Robots-Tag disagree

When both are present the most restrictive rule wins, which is rarely what someone had in mind when they set them in two places. Pick one mechanism per page type and apply it through templates rather than by hand — a governance problem more than a technical one, and one worth writing into your SEO governance rules.

Shipping robots.txt changes untested

One stray disallow: / can remove a site from search. Note that the old Search Console robots.txt Tester was retired in November 2023; the current tool is the robots.txt report, which shows the files Google fetched for your top hosts, when it fetched them, and any parse errors. Pair it with URL Inspection to check individual URLs.

Should I Use Robots.txt or Noindex? Deciding in 30 Seconds

Most of the robots.txt vs meta robots decision collapses into one question: do you want the page fetched, or do you want it out of the results? That is the difference between crawling and indexing, and it picks the tool for you. Robots.txt stops the fetch. Meta robots noindex stops the listing. They are not interchangeable, and reaching for the wrong one is the most common reason a page you wanted gone stays visible.

The short version of noindex vs disallow: a Disallow rule tells Google not to request the URL at all, so Google never reads anything on it — including a noindex you put there. A noindex rule lets Google fetch the page but keeps it out of the index. The question underneath all of this is what does noindex do that a disallow rule cannot, and the answer is short: it removes a page that is already ranking.

So, does robots.txt stop indexing? No. A disallowed URL can still surface as a bare link with no snippet when other pages link to it, because Google can index a URL it has never fetched. And can you noindex in robots.txt? No — Google stopped honouring unsupported noindex lines in robots.txt on 1 September 2019, and such a line is silently ignored today.

The rule of thumb, in the order you will actually need it:

• The page must disappear from search: leave it crawlable, add meta robots noindex, and wait for a recrawl.
• The page must never be fetched (crawl budget, infinite parameter space, faceted URL explosion): Disallow it in robots.txt, and accept that it may still appear as a URL-only result.
• The file is not HTML (PDF, image, CSV, plain text): there is no <head> to hold a tag, so use X-Robots-Tag in the HTTP response header.
• You need both: noindex first, confirm the page has dropped out of results, then disallow. Never the other way round.

Robots.txt vs Meta Robots: Frequently Asked Questions

Where Crawl Directives Fit in a Technical SEO Programme

Crawl directives are one control surface among several. They interact with canonicalisation, XML sitemaps, redirect policy and crawl budget, and a change to one usually implies a change to another — which is why directive problems are best audited as a set rather than one tag at a time. The same discipline applies to templated content that generates directives automatically.

Auditing that whole surface is part of our technical SEO service. If you would rather see the current state of your own directives first, start with a free SEO audit. Realistic timelines for remediation of this kind are covered in how long SEO takes for results, and the metrics worth watching afterwards in SEO KPIs and reporting. For the wider discipline this sits inside, start with our guide to technical SEO.

Sources

Directive behaviour on this page is checked against Google's own documentation: How Google interprets the robots.txt specification and Robots meta tag, data-nosnippet and X-Robots-Tag specifications. Last verified 11 August 2026.