These two tools sound almost identical, get confused constantly, and control completely different things. Mix them up and you can accidentally keep a page out of Google — or leave a private page fully indexed. Here’s the plain-English difference between robots.txt and the meta robots tag, when to reach for each, and the one mistake that trips up almost everyone.
The one-sentence difference
Robots.txt controls crawling (whether a bot is allowed to fetch a page). The meta robots tag controls indexing (whether a page that’s been fetched is allowed to appear in search results). Crawling and indexing are two separate stages, and each tool governs a different one.
What robots.txt does
Robots.txt is a single file at the root of your domain that tells crawlers which paths they may or may not request. It’s a crawl instruction, issued before the bot ever loads the page:
User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
Key point: Disallow stops a bot from crawling a URL — it does not guarantee the URL stays out of Google. If other sites link to a disallowed page, Google can still index the URL (usually with no description, because it was never allowed to read the content). If you need a hosted robots.txt fast, our Robots.txt Generator builds one in your browser, and the complete robots.txt guide covers every directive.
What the meta robots tag does
The meta robots tag lives in the <head> of an individual page and tells search engines what to do after they’ve crawled it. The most common use is keeping a page out of the index:
<meta name="robots" content="noindex, follow">
This says: don’t show this page in search results, but do follow its links. Because it’s a per-page instruction that Google reads directly, noindex is the reliable way to keep a page out of search — unlike a robots.txt Disallow.
Common values:
index, follow— the default; show the page and follow its links.noindex, follow— hide the page, but still crawl its links (great for thank-you pages, filtered listings, thin archives).noindex, nofollow— hide the page and ignore its links.noindexcan also be sent as an HTTP header (X-Robots-Tag) for non-HTML files like PDFs.
The mistake almost everyone makes
Here’s the trap: to reliably noindex a page, Google must be able to crawl it and read the meta robots tag. If you Disallow that same page in robots.txt, Google never fetches it, never sees the noindex, and the URL can linger in search results anyway.
# robots.txt Disallow: /private-page/ # /private-page/ <head> <meta name="robots" content="noindex"> # Google never crawls it, never sees noindex → may stay indexed
Rule of thumb: if you want a page gone from Google, use noindex and make sure robots.txt allows crawling of it. Only use robots.txt Disallow to save crawl budget on sections you don’t care about indexing at all.
When to use which
| Goal | Use |
|---|---|
| Keep a page out of Google reliably | Meta robots noindex (allow crawling) |
| Stop bots wasting crawl budget on admin/cart/search URLs | robots.txt Disallow |
| Hide a page but keep passing link equity | noindex, follow |
| Block a whole folder from crawling | robots.txt Disallow: /folder/ |
| Keep a PDF or image out of search | X-Robots-Tag: noindex header |
| Point crawlers to your sitemap | robots.txt Sitemap: line |
How they work together
Used correctly, they’re a team: robots.txt keeps crawlers out of the areas that would waste their time (admin, cart, faceted URLs), while meta robots precisely controls which of your crawlable pages actually show up in search. Neither one forces Google to index a page — that’s always Google’s call — but together they give clear crawl and index signals.
The bottom line
Remember the split: robots.txt = crawling, meta robots = indexing. To remove a page from search, reach for noindex and leave it crawlable. To conserve crawl budget on junk URLs, reach for robots.txt Disallow. Never block a page in robots.txt and expect a noindex on it to work — that’s the contradiction that quietly leaves private pages in Google.
Frequently asked questions
Does robots.txt Disallow remove a page from Google?
Not reliably. Disallow stops crawling, but if the URL is linked from elsewhere, Google can still index it (often without a description). To remove a page from search, use a meta robots noindex tag and keep the page crawlable.
Can I use both robots.txt and meta robots on the same page?
You can, but be careful: if robots.txt disallows the page, Google can’t crawl it and therefore can’t read its noindex tag. For a noindex to work, the page must be crawlable.
What’s the difference between noindex and nofollow?
noindex keeps the page out of search results. nofollow tells search engines not to follow the links on that page. They’re independent — noindex, follow is a common combination.
Where does the meta robots tag go?
In the <head> section of the individual page’s HTML: <meta name="robots" content="noindex, follow">. For non-HTML files like PDFs, use the X-Robots-Tag HTTP header instead.
Which should I use to save crawl budget?
robots.txt. Disallowing large, low-value sections (admin, internal search, faceted URLs) stops crawlers wasting requests there. Meta robots doesn’t save crawl budget because the page still has to be crawled to read the tag.
Related tools & guides
- Robots.txt Generator — build a robots.txt file in your browser.
- The Complete Robots.txt Guide (2026) — every directive explained.
- Robots.txt for WordPress — platform-specific setup.
- The Complete XML Sitemap Guide (2026) — tell Google what to crawl.
Handy tools for this topic
70+ free tools, zero sign-up
Every ToolBrainy tool runs right in your browser — no accounts, no watermarks and no limits. Compress a PDF, generate a strong password, convert an image and plenty more.
