If you run a WordPress site, there is a good chance you have never actually seen your robots.txt file — and yet it is quietly shaping how Google crawls every page you publish. WordPress generates one for you automatically, in memory, and most people never touch it. That is fine until the day you need to change it and discover there is no file to edit.
This guide walks through exactly how robots.txt works on WordPress: where it lives, what the default rules do, a safe template you can copy, the handful of mistakes that quietly tank crawling, and how to check your work. No fluff, just the things that matter for a real WordPress site.
What robots.txt actually does on WordPress
A robots.txt file sits at the root of your domain (https://yoursite.com/robots.txt) and tells search engine crawlers which parts of your site they are welcome to request. It is the first file Googlebot looks for when it visits.
Two things are worth burning into memory before you touch anything:
- Robots.txt controls crawling, not indexing. Blocking a URL here stops bots from fetching it — but a blocked URL can still appear in search results if other pages link to it. To keep something out of the index, you need a
noindexmeta tag, not a Disallow rule. - It is a public file. Anyone can read yours. Never use it to “hide” login pages, private folders, or sensitive URLs — you are just handing out a map.
The virtual file gotcha (this trips up everyone)
Here is the part unique to WordPress. By default there is no physical robots.txt file on your server. WordPress creates a virtual one on the fly every time a bot requests it, using the do_robots hook. If you FTP into your site and look in the root folder, you will find nothing named robots.txt — but the URL still works.
This matters because it changes how you edit it:
- If no physical file exists, WordPress serves its virtual default (shown below).
- The moment you upload a real robots.txt file to the site root, WordPress stops generating the virtual one and serves your file instead. Your file wins.
- Most SEO plugins (Yoast, Rank Math, All in One SEO) let you edit the virtual file through the dashboard by filtering that same hook — so you get a real editor without touching FTP.
What WordPress puts in the default robots.txt
Out of the box, a WordPress site serves something close to this:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yoursite.com/wp-sitemap.xml
It is intentionally minimal, and honestly it is close to correct for most sites. It blocks the admin area, keeps admin-ajax.php open (many themes and plugins need it for front-end features), and points to the core sitemap. If your whole SEO setup is “leave it alone,” you could do a lot worse.
A better robots.txt template for WordPress
For a typical content or business site, this is a clean, safe starting point. Adjust the sitemap URL to match yours.
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /search/ Disallow: /*?replytocom Disallow: /trackback/ Sitemap: https://yoursite.com/wp-sitemap.xml
What the extra lines do: /?s= and /search/ stop crawlers from wasting crawl budget on internal search result pages (which are thin and infinite). ?replytocom and /trackback/ block the low-value comment-reply and trackback URLs WordPress generates. None of this blocks a single real page or post.
Rather than hand-editing, you can build a rule set for your exact setup — WordPress, WooCommerce, or a custom stack — with our free Robots.txt Generator. It writes valid syntax for you and adds the sitemap line automatically.
What you should NOT block on WordPress
This is where most robots.txt damage happens. A few Disallow lines that look tidy will actively hurt your rankings:
- Never block
/wp-content/. Your images, and often your CSS and JavaScript, live here. If Google cannot fetch your CSS/JS, it cannot render your pages properly and may judge them as broken or non-mobile-friendly. - Never block
/wp-includes/wholesale. Older “SEO hardening” guides told you to. Modern Google explicitly asks you not to — it needs those assets to render. - Do not block
/wp-content/uploads/. That is your media library. Blocking it kills Google Images traffic. - Do not block your category or tag archives here if you want them indexed. If you want them out of the index, use a
noindextag (via your SEO plugin) instead — see the distinction below.
Blocking specific bots
You can set rules per crawler. A common example is allowing Google fully while slowing down or blocking a scraper:
User-agent: Googlebot Allow: / User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Disallow: /
A word of caution: well-behaved bots (Google, Bing) respect these rules; malicious scrapers ignore robots.txt entirely. Use it for crawl management, not security. And in 2026, many site owners also add explicit rules for AI crawlers like GPTBot and PerplexityBot — allow or disallow them based on whether you want your content used for AI training and answers.
WooCommerce and plugin considerations
If you run a store, WordPress and WooCommerce generate URLs you almost certainly do not want crawled:
Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /*add-to-cart=*
The cart, checkout, and account pages are user-specific and have no search value, and the add-to-cart query strings create endless duplicate URLs. Blocking them keeps crawlers focused on your product and category pages, where the traffic actually is.
Common WordPress robots.txt mistakes
- Leaving a
Disallow: /from a staging site. Developers block everything during a build. If that line survives launch, you have told Google to crawl nothing. This is the single most common way a new WordPress site gets zero traffic. - Using robots.txt to deindex. Blocking a page here does not remove it from Google — it just stops Google from seeing the
noindextag on it. To remove a page, allow crawling and addnoindex. - Blocking CSS/JS. Covered above, but it is worth repeating: it breaks rendering.
- Forgetting the sitemap line. It is a free, direct signal to crawlers. Always include it.
- Editing the wrong file. If your SEO plugin manages a virtual robots.txt but you also uploaded a physical one, the physical file wins and your plugin edits do nothing.
Robots.txt vs noindex: which to use
This is the distinction that fixes 90% of confusion. They solve different problems:
Goal: Save crawl budget on junk URLs → robots.txt Disallow
Goal: Keep a page out of Google → noindex meta tag
Goal: Both → allow crawl + noindex
(NOT Disallow — the bot must
be able to see the noindex)
Rule of thumb: if you never want a bot to fetch it, use robots.txt. If you are fine with fetching but want it kept out of search results, use noindex. If you want a page gone from Google, do not block it in robots.txt, because then Google can’t read the noindex tag telling it to leave.
How to test your robots.txt
After any change, verify it:
- Load it directly: visit
https://yoursite.com/robots.txtin a browser. What you see is what bots see. - Google Search Console: the robots.txt report shows the fetched version, when it was last crawled, and any parse errors.
- Spot-check a URL: use the URL Inspection tool in Search Console to confirm an important page is not accidentally blocked.
- Confirm your sitemap resolves: the URL in your
Sitemap:line should return a valid XML sitemap, not a 404.
Two of our tools pair naturally with this: build the file with the Robots.txt Generator, then generate the matching sitemap with the XML Sitemap Generator so the two line up.
Frequently asked questions
Does WordPress have a robots.txt file by default?
Yes, but it is virtual — generated in memory, not saved as a file on your server. Visiting yoursite.com/robots.txt shows it. As soon as you upload a real file to the site root, WordPress serves that instead.
How do I edit robots.txt in WordPress without FTP?
Use an SEO plugin. Yoast, Rank Math, and All in One SEO all include a robots.txt editor in the dashboard that modifies the virtual file for you — no FTP or file manager needed.
Should I block wp-admin in robots.txt?
WordPress already does, and that is fine — but always keep the Allow: /wp-admin/admin-ajax.php line, because many themes and plugins use that endpoint for front-end functionality that Google needs to render.
Will robots.txt remove a page from Google?
No. Blocking a URL stops crawling, but the page can still be indexed from external links. To remove a page, allow crawling and add a noindex meta tag so Google can see the instruction to drop it.
Do I need a robots.txt file at all?
Not strictly — a site with no robots.txt is crawled normally. But a good one saves crawl budget on junk URLs and points bots to your sitemap, so it is worth having on any WordPress site.
Where do I add the sitemap in robots.txt on WordPress?
Add a line Sitemap: https://yoursite.com/wp-sitemap.xml (or your plugin’s sitemap URL) anywhere in the file — it is not tied to a User-agent block. Point it at whichever sitemap your site actually serves.
The short version
For most WordPress sites, the default robots.txt is nearly right — the wins come from a few careful additions (block internal search and store-only URLs, always include your sitemap) and, more importantly, from not making the classic mistakes: never block CSS/JS or /wp-content/, never leave a staging Disallow: / in place, and never confuse robots.txt with noindex. Get those right, verify in Search Console, and you can safely forget about the file for years.
Ready to build yours? Generate a valid file in seconds with the Robots.txt Generator, pair it with the XML Sitemap Generator, and if you want the full background, read The Complete Robots.txt Guide 2026.
Handy tools for this topic
70+ free tools, zero sign-up
Every ToolBrainy tool runs right in your browser — no accounts, no watermarks and no limits. Compress a PDF, generate a strong password, convert an image and plenty more.
