Create a custom robots.txt file for your website in seconds. This free robots.txt maker lets you set crawler rules, allow and disallow paths, and add your sitemap URL — then generate and download the file instantly. No signup.
* to target all bots.
The Robots.txt Generator lets you create a properly formatted robots.txt file for your website without writing any code. Simply choose the search-engine bots you want to configure, add the paths you want to block or allow, and include your sitemap URL. The live preview updates instantly so you can see exactly what the file will look like before you download it. Everything runs inside your browser with no data sent to any server.
Reviewed by the ToolBrainy Team · Runs entirely in your browser · Last updated July 2026
Select a user-agent such as * (all bots) or Googlebot from the dropdown.
Click "Add path" under Disallow or Allow to control which pages crawlers can visit.
Copy the result or click "Download robots.txt" and upload it to the root of your site.
Configure separate rules for Googlebot, Bingbot, GPTBot, and any custom crawler.
See the final robots.txt format update in real time as you make changes.
Add one or more sitemap URLs so search engines can discover your content quickly.
Your crawl rules are generated locally and never uploaded — nothing about your site is stored on our servers.
Prevent crawlers from indexing /admin, /login or /dashboard URLs.
Add GPTBot and ChatGPT-User blocks to keep your content out of AI training data.
Disallow everything with a wildcard rule to prevent staging URLs from appearing in search.
Include your sitemap URL so Google and Bing discover new pages faster.
Allow Googlebot full access while blocking Bingbot from specific sections.
Block cart and checkout pages to focus crawl budget on product and category pages.
Robots.txt is a small plain-text file that lives in the root folder of your website, at an address like https://yourdomain.com/robots.txt. Its job is to tell search-engine crawlers which parts of your site they are welcome to visit and which parts they should leave alone. When a crawler such as Googlebot arrives, checking this file is usually the first thing it does.
The file follows the Robots Exclusion Protocol, a long-standing agreement that the major search engines respect. Each robots.txt file is built from one or more groups of rules. A group starts by naming a crawler with a User-agent line, then lists the paths that crawler may or may not request. It is a set of polite instructions rather than a lock, so it works best for guiding well-behaved bots, not for keeping secrets.
Before a crawler fetches your pages, it downloads your robots.txt file and reads the rules that apply to it. Each rule is a single line made up of a directive and a value. Here are the directives you will actually use:
* means every bot.Disallow: / blocks the whole site.A crawler matches paths from left to right, and the most specific rule wins. In the example below, the whole /private/ folder is closed off, but one file inside it stays open.
User-agent: * Disallow: /private/ Allow: /private/public-notice.html Sitemap: https://yourdomain.com/sitemap.xml
One line worth remembering: robots.txt controls crawling, not indexing. It asks a bot not to fetch a page. If you need a page kept out of search results, use a noindex meta tag instead, which we explain further down.
Search engines give every site a rough crawl budget — the number of pages they will fetch in a given period. A tidy robots.txt file helps them spend that budget on the pages you actually want ranked. This matters more as a site grows and picks up thin or duplicate URLs that add nothing to search results.
Used carefully, robots.txt is a quiet SEO win. Used carelessly, a single wrong line can hide your best pages, so it pays to check your rules before you publish them.
Most robots.txt problems come from a few repeat offenders. Watch out for these:
Disallow: / left over from a staging setup can wipe you out of search./wp-includes/ or asset folders can hurt how your pages are understood.noindex, not Disallow./Folder is not /folder. A trailing slash also changes the match.Real files are usually short. Here are the patterns you will reach for most often — copy one as a starting point and adjust the paths to match your site.
Allow every crawler everywhereUser-agent: * Disallow:Block the whole site (useful for staging)
User-agent: * Disallow: /Block a few folders but allow the rest
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /search/ Sitemap: https://yourdomain.com/sitemap.xmlAllow Googlebot, block a specific bot
User-agent: Googlebot Disallow: User-agent: AhrefsBot Disallow: /Keep an AI scraper out of your content
User-agent: GPTBot Disallow: / User-agent: * Disallow:
WordPress creates a virtual robots.txt automatically, so a fresh install already has one even if there is no file on the server. It is basic, but for most sites it is enough. If you want more control, you can replace it with your own rules. A sensible starting point for a WordPress site looks like this:
robots.txt for WordPressUser-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yourdomain.com/wp-sitemap.xml
Notice that /wp-admin/ is blocked while admin-ajax.php stays open, because some themes and plugins rely on it to load content. Avoid blocking /wp-includes/ or your uploads folder — doing so can stop Google from rendering your pages properly. If you use an SEO plugin like Yoast, Rank Math or All in One SEO, each one has a built-in robots.txt editor, so you can paste these rules straight into the dashboard without touching FTP.
Sitemap line helps search engines find everything worth crawling.Once you have generated your rules above, getting the file live takes only a minute. The exact steps depend on how you manage your site:
robots.txt — all lowercase, nothing extra.public_html) so it opens at yourdomain.com/robots.txt.yourdomain.com/robots.txt in your browser. If your rules show up, you are done.Testing takes seconds and catches mistakes before search engines act on them. A few quick ways to check your file:
yourdomain.com/robots.txt and read the rules. It should return the file, not a 404.A robots.txt file sits at the root of your website and tells search-engine crawlers which pages or sections they are allowed or not allowed to visit. It follows the Robots Exclusion Standard and is read by bots before they crawl your site.
Well-behaved crawlers like Googlebot and Bingbot respect robots.txt rules. However, it is not a security measure. Malicious bots may ignore the file entirely. Use server-level access controls or password protection for truly private content.
The asterisk * is a wildcard that matches all crawlers. Rules under this block apply to every bot that reads the file, unless a more specific user-agent block overrides them.
Place the file at the root of your domain, for example https://yourdomain.com/robots.txt. It must be accessible at exactly that URL for crawlers to find it.
Yes. Add a dedicated block that starts with User-agent: Googlebot and give it its own Allow and Disallow lines. Google's crawler follows the most specific matching block, so those rules apply to Googlebot while your User-agent: * block still covers every other crawler. This tool lets you add per-crawler blocks so you can, for example, let Googlebot into a folder you block for everyone else.
Build your rules with the generator above, then in your Blogger dashboard go to Settings → Crawlers and indexing → Enable custom robots.txt, and paste the generated file. A custom robots.txt for Blogger (Blogspot) lets you control which pages Googlebot crawls, block search and label pages, and point crawlers to your sitemap for faster, cleaner indexing.
WordPress serves a virtual robots.txt by default. To use a custom one, generate your rules here and either upload a robots.txt file to your site root via FTP, or paste the rules into your SEO plugin — Yoast, Rank Math and AIOSEO all include a robots.txt editor. This gives you full control over crawling rules and sitemap discovery for your WordPress site.
No. A Disallow rule only asks a crawler not to fetch the page. If other sites link to that URL, it can still show up in results, often without a description. To keep a page out of search, add a noindex meta tag or use the removal tool in Search Console, and make sure the page is not blocked so Google can actually read the noindex.
Crawlers cache the file, so changes are not picked up instantly — Google typically refreshes it within about a day. Because of this, always confirm your new version is live and give it a little time before assuming a rule has taken effect. Google also reads only the first 500KB of the file, which is far more than a normal robots.txt needs.
Robots.txt works at the site level and controls crawling — whether a bot may request a URL at all. A meta robots tag (or the X-Robots-Tag header) works at the page level and controls indexing — whether a page may appear in search. Use robots.txt to manage crawl budget, and use noindex when you specifically want a page kept out of results.