Free Robots.txt Generator and Tester

About this tool

This tool does three jobs that usually need three separate tools. The Generate tab builds a properly formatted robots.txt file without writing any code — choose the bots you want to configure, add the paths to block or allow, and include your sitemap URL, with a live preview that updates as you type. The AI Crawlers tab turns the confusing world of GPTBot, ClaudeBot, PerplexityBot and Google-Extended into a set of deliberate choices, telling you in plain English what each block actually costs you. The Test & Validate tab checks any URL against any site's rules and shows you the exact line that decides the outcome.

The builder, the AI crawler panel and the rule matcher all run inside your browser — nothing you type is uploaded. The one exception is the "Load a live site's robots.txt" button: browsers are not allowed to read another domain's files directly, so that request passes through our server. Only the domain you enter is sent, and the fetched file is cached for five minutes.

Rule matching runs in your browser · Updated September 24, 2026

How to create and test your robots.txt file

1. Read your current file first. Open https://yourdomain.com/robots.txt and save a copy before replacing it. Your CMS or SEO plugin may already manage it.
2. Choose a crawler. Keep * for general rules, or add a separate user-agent block for a specific bot. Remove the sample /admin/ rule if it does not apply to your site.
3. Add only the paths you need to control. Use Disallow to restrict crawling and Allow for a more specific exception. Leave public articles, images, CSS and JavaScript accessible.
4. Add your actual sitemap URL. Open it first to confirm it works. The example domain below must be replaced with your own.
5. Test before downloading. Open Test & Validate, choose Use my rules, enter paths on separate lines and select the crawler. Check both blocked paths and pages that should remain accessible.
6. Publish and check again. Download the file and place it at your site's root, or update the file through your CMS. Open the public file and use Search Console URL Inspection to check an important page.

Generating or testing a file here does not change your website. Robots.txt controls crawling; it does not guarantee indexing or higher rankings.

Try this example in the robots.txt tester

Paste these rules into Test & Validate and choose Googlebot. This example blocks a folder while keeping one file in that folder crawlable.

User-agent: *
Disallow: /drafts/
Allow: /drafts/public-example.html

Sitemap: https://example.com/sitemap.xml

• /drafts/unfinished.html: blocked by the folder rule.
• /drafts/public-example.html: allowed because the matching Allow path is more specific.
• /blog/my-post/: allowed because no Disallow rule matches it.

These are crawl permissions, not privacy protections. Keep confidential drafts behind authentication. A crawler can still discover a blocked URL through links.

Why use it

Use cases

What is robots.txt?

Robots.txt is a small plain-text file that lives in the root folder of your website, at an address like https://yourdomain.com/robots.txt. Its job is to tell search-engine crawlers which parts of your site they are welcome to visit and which parts they should leave alone. When a crawler such as Googlebot arrives, checking this file is usually the first thing it does.

The file follows the Robots Exclusion Protocol, a long-standing agreement that the major search engines respect. Each robots.txt file is built from one or more groups of rules. A group starts by naming a crawler with a User-agent line, then lists the paths that crawler may or may not request. It is a set of polite instructions rather than a lock, so it works best for guiding well-behaved bots, not for keeping secrets.

How robots.txt works

Before a crawler fetches your pages, it downloads your robots.txt file and reads the rules that apply to it. Each rule is a single line made up of a directive and a value. Here are the directives you will actually use:

• User-agent — names the crawler a group of rules applies to. * means every bot.
• Disallow — a path that the named crawler should not request. Disallow: / blocks the whole site.
• Allow — an exception that opens a path back up, even inside a blocked folder.
• Sitemap — the full URL of your XML sitemap so crawlers can find your pages faster.

A crawler matches paths from left to right, and the most specific rule wins. In the example below, the whole /private/ folder is closed off, but one file inside it stays open.

User-agent: *
Disallow: /private/
Allow: /private/public-notice.html

Sitemap: https://yourdomain.com/sitemap.xml

One line worth remembering: robots.txt controls crawling, not indexing. It asks a bot not to fetch a page. If you need a page kept out of search results, use a noindex meta tag and keep the page crawlable. Read our guide to meta robots versus robots.txt to choose the right instruction.

Why robots.txt matters for SEO

Search engines give every site a rough crawl budget — the number of pages they will fetch in a given period. A tidy robots.txt file helps them spend that budget on the pages you actually want ranked. This matters more as a site grows and picks up thin or duplicate URLs that add nothing to search results.

• Focus crawl budget. Keep bots away from cart, filter and internal search pages so they reach your real content sooner.
• Reduce server load. Blocking heavy or endless URL patterns stops aggressive crawling from slowing your site down.
• Point to your sitemap. A sitemap line helps engines discover new and updated pages quickly.
• Cut duplicate crawling. Steering bots away from tag archives and print versions reduces wasted requests.

Used carefully, robots.txt is a quiet SEO win. Used carelessly, a single wrong line can hide your best pages, so it pays to check your rules before you publish them.

Set AI crawler rules deliberately

Training crawlers, search crawlers and user-requested fetchers have different roles. Review each provider's current documentation before applying a preset. Allowing a bot does not guarantee citations or referral traffic, and user-requested fetches may be treated differently from automatic crawling.

Robots.txt is voluntary, not an access-control system. Rules affect compliant future requests; they do not remove previously collected content.

Common robots.txt mistakes

Most robots.txt problems come from a few repeat offenders. Watch out for these:

• Blocking the entire site by accident. A stray Disallow: / blocks crawling and can prevent Google from reading updated content. It does not reliably remove indexed URLs.
• Expecting it to hide private data. Anyone can open your robots.txt, so listing a secret folder there simply advertises it. Use passwords or server rules for anything sensitive.
• Blocking CSS and JavaScript. Google needs these files to render your pages. Blocking /wp-includes/ or asset folders can hurt how your pages are understood.
• Assuming Disallow removes a page. A blocked URL can still appear in results if other sites link to it. For an accessible page you want excluded, use noindex and allow Google to crawl it so it can read the instruction.
• Case and slash slips. Paths are case-sensitive, and /Folder is not /folder. A trailing slash also changes the match.
• Putting the file in the wrong place. It only works at the domain root, never in a subfolder.

Robots.txt examples

Real files are usually short. Here are the patterns you will reach for most often — copy one as a starting point and adjust the paths to match your site.

User-agent: *
Disallow:

User-agent: *
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/

Sitemap: https://yourdomain.com/sitemap.xml

User-agent: Googlebot
Disallow:

User-agent: AhrefsBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow:

WordPress robots.txt example

Before replacing your site's file, follow our WordPress robots.txt guide to understand virtual files, crawler rules and sitemap settings.

WordPress creates a virtual robots.txt automatically, so a fresh install already has one even if there is no file on the server. It is basic, but for most sites it is enough. If you want more control, you can replace it with your own rules. A sensible starting point for a WordPress site looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/wp-sitemap.xml

Notice that /wp-admin/ is blocked while admin-ajax.php stays open, because some themes and plugins rely on it to load content. Avoid blocking /wp-includes/ or your uploads folder — doing so can stop Google from rendering your pages properly. If you use an SEO plugin like Yoast, Rank Math or All in One SEO, each one has a built-in robots.txt editor, so you can paste these rules straight into the dashboard without touching FTP.

Robots.txt best practices

• Keep it simple. Only add rules you understand. A short, clear file is easier to maintain and less likely to break.
• Always include your sitemap. One Sitemap line helps search engines find everything worth crawling.
• Never block CSS, JS or images you want rendered. Blocking them can change how your pages are judged.
• Use noindex for pages you want out of search. Keep those pages crawlable so Google can read the noindex instruction.
• Match paths carefully. Remember that rules are case-sensitive and that a trailing slash matters.
• Test before and after every change. A quick check saves you from a costly mistake.
• Review it now and then. As your site changes, old rules can quietly block the wrong things.

How to upload robots.txt

Once you have generated your rules above, getting the file live takes only a minute. The exact steps depend on how you manage your site:

• Save the file. Copy your rules into a new plain-text file named exactly robots.txt — all lowercase, nothing extra.
• Upload it to the root. Using FTP or your hosting file manager, place it in the top-level folder (often public_html) so it opens at yourdomain.com/robots.txt.
• On WordPress, use a plugin. Yoast, Rank Math and AIOSEO each include a robots.txt editor, so you can paste and save your rules without any file transfer.
• Confirm it works. Visit yourdomain.com/robots.txt in your browser. If your rules show up, you are done.

How to test robots.txt

Google retired its robots.txt Tester in December 2023. The report that replaced it, inside Search Console, only shows the last version Google fetched for a property you have verified — it dropped the interactive "does this URL match these rules" check entirely, and you can no longer test another site's file at all. That is the gap the Test & Validate tab above fills.

Testing takes seconds and catches mistakes before search engines act on them. A few ways to check your file:

• Use the tester above. Paste your rules or fetch any live domain, drop in a list of URLs, pick a crawler, and every URL is marked allowed or blocked with the exact line that decided it. It follows Google's matching rules: the longest matching path wins, and a tie goes to Allow.
• Open it directly. Load yourdomain.com/robots.txt and read the rules. It should return the file, not a 404.
• Use Google Search Console. The URL Inspection tool still tells you whether one specific page on your own verified property is blocked, and the robots.txt report shows fetch errors.
• Check a competitor's file. Fetching another site's robots.txt shows you which sections they keep out of search and which AI crawlers they allow — often a faster read on their strategy than any tool.
• Re-check after every edit. Search engines cache the file, so confirm the new version is live and test the paths you care about.

Check your rules against Google's documentation

For a walkthrough of the file format and common rules, start with our robots.txt guide. Use Google's documentation below to check how Google interprets those rules.

• What robots.txt can and cannot do
• How Google matches robots.txt rules
• How noindex prevents indexing

This tester is a preview of rule matching, not a live Google crawl. Confirm important changes with Search Console after publishing.

Frequently asked questions

Related tools