The first time robots.txt cost me traffic, I didn’t even know it happened. A staging site went live with a single leftover line — Disallow: / — and for almost two weeks Google politely stayed away from the entire thing. No error, no warning, just a slow, confusing slide in impressions until I finally opened the file and felt my stomach drop.
That’s the strange thing about robots.txt. It’s one of the smallest files on your website — often just a few lines of plain text — yet it sits at the very front door of how search engines see you. Get it right and it quietly does its job for years. Get one character wrong and you can hide your whole site from Google without realizing it.
This guide walks through everything, from the absolute basics to the advanced edge cases. Whether you run a WordPress blog, a Shopify store, or a custom-built app, by the end you’ll understand exactly what to put in your robots.txt file, what to keep out of it, and how to check your work before it goes live.
What is robots.txt?
Robots.txt is a plain text file that lives in the root folder of your website. Its job is to tell automated visitors — search engine crawlers, bots, and scrapers — which parts of your site they’re allowed to request and which parts they should leave alone.
When a well-behaved crawler like Googlebot arrives at your site, the first thing it does is look for a file at a very specific address:
https://yourdomain.com/robots.txt
If it finds one, it reads the rules before crawling anything else. If it doesn’t find one, it assumes everything is fair game and crawls whatever it can reach. That’s an important point people miss: no robots.txt file means “crawl everything,” not “crawl nothing.”
The name is short for “robots exclusion” and the format is governed by the Robots Exclusion Protocol, a standard that’s been around since 1994 and was finally formalized by the IETF in 2022. Every major search engine respects it.
Robots.txt is a request, not a lock. Good bots obey it. Malicious bots and scrapers can — and do — ignore it completely. Never use robots.txt to hide sensitive information; use passwords and proper access control for that.
Why robots.txt matters for SEO
You might be wondering why you’d ever want to stop search engines from crawling parts of your site. Isn’t more crawling better? Not always. Here’s why the file matters.
It protects your crawl budget
Search engines don’t crawl every page on every site on every visit. They allocate a rough budget of how many pages they’ll fetch from you in a given window. If a big chunk of that budget gets spent on pages that don’t matter — internal search results, filter combinations, admin URLs — your genuinely important pages get crawled less often. Robots.txt lets you steer crawlers away from the low-value stuff.
It keeps clutter out of the crawl path
Most sites generate a surprising number of throwaway URLs: session parameters, faceted navigation, print versions, staging directories. Blocking these keeps crawlers focused on the content you actually want ranked.
It points crawlers to your sitemap
Robots.txt is also the conventional place to announce where your XML sitemap lives, which helps search engines discover your important pages faster. We’ll cover that pairing in detail later.
It prevents server strain
On large or resource-heavy sites, aggressive crawling can slow things down for real visitors. Thoughtful robots.txt rules reduce unnecessary requests.
The flip side is the danger I opened with: because robots.txt is so powerful, a careless rule can deindex pages you desperately want ranked. Respect it, test it, and change it deliberately.
Robots.txt syntax explained
The good news is that the syntax is genuinely simple. A robots.txt file is a list of rules, and each rule has two parts: who the rule applies to, and what they can or can’t do.
Here is the smallest useful example:
User-agent: *
Disallow: /private/
Read out loud, that says: “To every crawler (* means all of them), you are not allowed to crawl anything inside the /private/ folder.” Everything else on the site is allowed by default.
A few rules of the road for the format itself:
- Each directive goes on its own line.
- Directive names are not case-sensitive (
User-agent and user-agent both work), but the paths you list are case-sensitive.
- Lines starting with
# are comments and are ignored by crawlers — use them to leave notes for yourself.
- Paths are relative to the root, always starting with a
/.
- A blank line separates one rule group from the next.
The common directives
You only need to understand four directives to handle 95% of real-world situations. Let’s take them one at a time.
User-agent
This line names the crawler a rule group applies to. The asterisk * is a wildcard meaning “all crawlers.” You can also target specific bots by name:
# Rules for everyone
User-agent: *
Disallow: /cart/
# A special rule just for Google's image crawler
User-agent: Googlebot-Image
Disallow: /private-photos/
Some crawler names worth knowing: Googlebot (Google’s main crawler), Bingbot (Microsoft Bing), Googlebot-Image, and increasingly the AI crawlers like GPTBot and ClaudeBot that fetch content for large language models.
Disallow
This tells the named crawler not to request URLs that begin with the given path. A few examples make the pattern clear:
Disallow: /admin/ # blocks the whole /admin/ folder
Disallow: /search # blocks any URL starting with /search
Disallow: /*.pdf$ # blocks every URL ending in .pdf
Disallow: # an empty Disallow blocks nothing — allows all
Notice the last one. An empty Disallow: value is the standard way to say “this crawler may access everything.”
Allow
Allow is the exception-maker. It’s most useful when you’ve blocked a whole folder but want to permit one specific thing inside it:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
That blocks the entire WordPress admin area except the one file that themes and plugins legitimately need crawlers to reach. When rules conflict, most modern crawlers follow the most specific (longest) matching path, so the Allow wins here.
Sitemap
The Sitemap directive announces the full URL of your XML sitemap. Unlike the others, it isn’t tied to any user-agent — it’s a standalone line, usually placed at the bottom of the file:
Sitemap: https://yourdomain.com/sitemap.xml
You can list more than one sitemap line if you have multiple sitemaps. Always use the full, absolute URL here, not a relative path.
A note on Crawl-delay
You may see Crawl-delay: 10 in older files, meant to tell a crawler to wait ten seconds between requests. Be aware that Google ignores this directive entirely — you control Google’s crawl rate in Search Console instead. Bing and a few others still honor it, so it’s not useless, just misunderstood.
WordPress robots.txt examples
WordPress is where most people meet robots.txt for the first time. By default WordPress generates a small virtual file for you, but the moment you want real control you’ll want your own. Here’s a sensible, safe starting point for a typical WordPress site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://yourdomain.com/wp-sitemap.xml
What each line is doing:
Disallow: /wp-admin/ keeps crawlers out of the dashboard.Allow: /wp-admin/admin-ajax.php re-opens the one file plugins need.Disallow: /?s= and /search/ stop internal search-result pages from being crawled — these are classic crawl-budget wasters.- The
Sitemap line points to WordPress’s built-in sitemap.
Notice what’s not here. You’ll sometimes find old tutorials telling you to block /wp-content/ or /wp-includes/. Don’t. Modern Google needs to fetch your CSS and JavaScript from those folders to render and understand your pages. Blocking them can actively hurt your rankings.
Shopify robots.txt examples
Shopify is a different story, and it trips people up. On Shopify you don’t have full free-form access to the file the way you do on a self-hosted site. Shopify generates a solid default robots.txt automatically — one that already blocks carts, checkout, and internal search — and for most stores you should simply leave it alone.
Since 2021, Shopify does let you customize it through a robots.txt.liquid template if you’re on a plan that allows theme code edits. A typical customization looks like adding or removing a rule inside that template rather than writing the file from scratch:
{% comment %} Block a collection filter that creates duplicate URLs {% endcomment %}
{%- if group.user_agent.value == '*' -%}
{{ 'Disallow: /collections/*?*sort_by*' }}
{%- endif -%}
Shopify’s default already handles the important exclusions — /cart, /checkout, /account, and search. The main reasons to touch it are blocking messy filtered collection URLs or removing a rule Shopify added that’s blocking something you actually want crawled. If you’re not comfortable editing Liquid, the safe move is to leave the default in place; it’s genuinely well-designed.
WooCommerce robots.txt examples
WooCommerce runs on top of WordPress, so you get full control again — and a store adds a few URLs worth managing. Cart, checkout, and account pages hold nothing a search engine needs to index, and blocking them keeps crawlers on your products and categories:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=*
Disallow: /?orderby=
Disallow: /?filter_*
Sitemap: https://yourstore.com/wp-sitemap.xml
The two lines to pay attention to are the parameter blocks. /*add-to-cart=* stops crawlers from following those “add to cart” action links, and /?orderby= and /?filter_* keep them out of the endless sorted-and-filtered variations of your shop pages. Those parameter URLs can multiply into thousands of near-identical pages — exactly the kind of thing that drains crawl budget on a busy store.
One caution specific to stores: never block /product/ or your shop and category pages. It sounds obvious written down, but it’s a mistake I’ve seen happen when someone copies an overly aggressive rule set from a forum.
Common mistakes and how to avoid them
Most robots.txt disasters come from a small handful of mistakes. Here are the ones worth burning into memory.
1. Accidentally blocking the whole site
The single most damaging line in existence:
User-agent: *
Disallow: /
That forward slash means “the root and everything under it.” It’s the exact line that hid my staging site. It belongs on development servers and nowhere else. If your traffic ever falls off a cliff for no obvious reason, this is the first thing to check.
2. Using robots.txt to hide a page from search results
This is the most common misconception in all of SEO. Blocking a URL in robots.txt stops crawling, but it does not guarantee the page stays out of Google’s index. If other sites link to that URL, Google can still list it in results — just without a description, showing “No information is available for this page.” To truly keep a page out of search, use a noindex meta robots tag and allow crawling so Google can see the tag. More on that difference below.
3. Blocking CSS and JavaScript
Google renders pages like a browser does. If you block the resources it needs to render — stylesheets, scripts, fonts — it sees a broken version of your page and may judge it poorly. Let crawlers reach your assets.
4. Getting case wrong
Paths are case-sensitive. Disallow: /Blog/ will not block /blog/. Match your real URLs exactly.
5. Forgetting the file after a site migration
When you move from staging to production, the staging robots.txt often comes along for the ride, complete with its Disallow: /. Migration checklists exist for a reason — put “review robots.txt” near the top of yours.
How to test a robots.txt file
Never publish a robots.txt change and just hope. Testing takes two minutes and saves weeks of grief.
- View the live file. Type your domain followed by
/robots.txt into a browser. What you see is exactly what crawlers see.
- Use Google Search Console. Search Console reports whether Google can fetch your robots.txt and flags syntax problems. The URL Inspection tool also tells you if a specific page is blocked from crawling.
- Test individual URLs. Before you trust a new rule, confirm that the URLs you meant to block are blocked — and, just as importantly, that the ones you meant to keep are still allowed.
- Validate the syntax. A misplaced character can silently break a rule. Running your file through a validator catches problems a quick glance misses.
If you’d rather not hand-write the file at all, you can generate a clean, correctly formatted one in a couple of clicks with the free ToolBrainy Robots.txt Generator — pick your rules, and it writes valid syntax for you.
Robots.txt vs meta robots
This is the distinction that separates people who understand crawling from people who guess at it. Robots.txt and the meta robots tag sound similar and are constantly confused, but they do genuinely different jobs.
| Robots.txt |
Meta robots tag |
| A file at your site root |
A tag inside a page’s HTML <head> |
| Controls crawling (whether a bot fetches the URL) |
Controls indexing (whether a fetched page appears in results) |
| Applies to files, folders, and URL patterns |
Applies to a single page |
| Cannot reliably remove a page from the index |
Can remove a page from the index with noindex |
Here’s the practical takeaway: to keep a page out of Google, use noindex, and do not block it in robots.txt. Those two instructions fight each other — if you block crawling, Google never fetches the page, so it never sees the noindex tag telling it to drop the page. The meta tag lives in the page’s head like this:
<meta name="robots" content="noindex, follow">
Use robots.txt to manage crawling at scale (whole folders, parameters, budget). Use meta robots to control the indexing of specific pages. You can check what meta tags a page is actually serving with the ToolBrainy Meta Tag Analyzer.
Robots.txt and XML sitemaps
Robots.txt and your XML sitemap are two halves of the same conversation. Robots.txt tells crawlers where not to go; the sitemap tells them where you’d most like them to go. They work best together.
Linking your sitemap from robots.txt is a small step with a real payoff — it helps search engines discover your important URLs without waiting to stumble across them through internal links:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xml
One rule that catches people out: never list a URL in your sitemap that you’ve blocked in robots.txt. You’d be sending a mixed signal — “please crawl this” and “don’t crawl this” about the same page. Keep the two files consistent. If you need to build or refresh your sitemap, the ToolBrainy Sitemap Generator produces a valid XML file you can point to from this exact directive.
Crawl budget considerations
“Crawl budget” is one of those phrases that sounds intimidating and turns out to be simple. It’s just the number of pages a search engine is willing to crawl on your site within a given timeframe, shaped by two things: how much crawling your server can handle without slowing down, and how much Google actually wants to crawl based on your site’s importance and freshness.
For most small sites — a few hundred pages — crawl budget is a non-issue. Google will happily crawl everything. It starts to matter when you have:
- Tens of thousands of URLs or more.
- Lots of auto-generated pages (faceted navigation, filters, calendars, internal search).
- A large store with countless product-variant and sort-order URLs.
On sites like those, robots.txt becomes a budget tool. By blocking the low-value, near-duplicate, and infinite-URL traps, you concentrate crawling on the pages that earn you traffic. Think of it as clearing the path so crawlers spend their limited time on your best work rather than wandering through filter combinations no human will ever search for.
A few habits that protect crawl budget: block internal search results, block obvious parameter clutter, keep your sitemap clean and current, and fix or redirect long chains of broken and redirected URLs so crawlers aren’t chasing dead ends.
Frequently asked questions
Where should the robots.txt file be located?
It must live in the root directory of your domain and be reachable at https://yourdomain.com/robots.txt. Crawlers only look in that exact spot — a robots.txt file placed in a subfolder is ignored. Each subdomain needs its own file too; blog.yourdomain.com uses its own robots.txt, separate from the main domain.
Do I even need a robots.txt file?
Not strictly. If you’re happy for search engines to crawl your entire site, you can skip it and nothing breaks. That said, having one is good practice — it lets you point to your sitemap, block low-value URLs, and gives you a place to make crawl decisions later. A simple, permissive file is better than none.
Will robots.txt keep my page out of Google?
No, not reliably. Blocking a URL stops crawling, but the page can still appear in search results if other sites link to it. To truly keep a page out of the index, use a noindex meta robots tag and allow crawling so Google can read that tag.
How often do search engines check robots.txt?
Google typically caches your robots.txt for up to 24 hours, so changes usually take effect within a day. If you’ve made an urgent fix, you can prompt a faster re-fetch through Google Search Console rather than waiting.
Can I block AI crawlers like GPTBot in robots.txt?
Yes. AI crawlers that respect the standard — such as GPTBot and ClaudeBot — read robots.txt like any other bot. Add a rule group naming the crawler and disallow what you want. Keep in mind this only works for bots that choose to obey; it isn’t an enforceable block.
What happens if my robots.txt has a syntax error?
Crawlers are fairly forgiving and will ignore lines they can’t parse, applying the rest. The danger is subtler: a malformed rule might silently fail to block what you intended, or an overly broad one might block more than you meant. That’s exactly why testing and validating before publishing is worth the two minutes.
Bringing it all together
Robots.txt rewards a little bit of care and punishes carelessness, which is why it’s worth understanding properly rather than copying a random file off a forum. Remember the essentials: it controls crawling, not indexing; an empty or missing file means “crawl everything”; a lone Disallow: / can hide your whole site; and it should always be kept in sync with your XML sitemap.
Start simple. Block the genuinely useless URLs — admin areas, internal search, cart and checkout on stores — point to your sitemap, and leave your real content and assets fully crawlable. Test the file, watch Search Console for a week, and adjust only when you have a specific reason to.
Do that, and this tiny text file becomes exactly what it’s meant to be: a quiet, reliable doorman that sends crawlers straight to your best pages and keeps them out of the clutter. If you’d rather skip the hand-editing, the free Robots.txt Generator builds a valid file for you in seconds, and pairing it with the Sitemap Generator, Meta Tag Analyzer, and SERP Preview gives you a complete, no-signup technical-SEO toolkit to get the rest right too.
